week-8/8-Conversation-and-Text-Generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 8 - Conversation and Text Generation\n",
    "Many natural language activities boil down to text generation, especially the back-and-forth nature of natural conversation and question answering. While some may regard it as a parlour trick due to unpredictability, recent dramatic improvements in text generation suggest that these kind of models can find themselves being used in more serious social scientific applications, such as in survey design and construction, idiomatic translation, and the normalization of phrase and sentence meanings.\n",
    "\n",
    "These models can be quite impressive, even uncanny in how human like they sound. Check out this [cool website](https://transformer.huggingface.co), which allows you to write with a transformer. The website is built by the folks who wrote the package we are using. The code underneath the website can be found in their examples: [run_generation.py](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py).\n",
    "\n",
    "Much 2022 NLP research is on text generation. Most famously, this is the primary use of large language models like GPT-3 (OpenAI), Wu Dao (Beijing Academy of AI), and Gopher (DeepMind)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Special module written for this class\n",
    "#This provides access to data and to helper functions from previous weeks\n",
    "#Make sure you update it before starting this notebook\n",
    "import lucem_illud #pip install -U git+https://github.com/UChicago-Computational-Content-Analysis/lucem_illud.git\n",
    "\n",
    "import sklearn #For generating some matrices\n",
    "import pandas as pd #For DataFrames\n",
    "import numpy as np #For arrays\n",
    "import matplotlib.pyplot as plt #For plotting\n",
    "import seaborn #Makes the plots look nice\n",
    "import seaborn as sns\n",
    "import scipy #Some stats\n",
    "import nltk #a little language code\n",
    "from IPython.display import Image #for pics\n",
    "\n",
    "import pickle #if you want to save layouts\n",
    "import os\n",
    "import io\n",
    "import zipfile\n",
    "\n",
    "import networkx as nx\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "import torch # pip install torch\n",
    "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from transformers import BertTokenizer, BertConfig # pip install tranformers\n",
    "from transformers import AdamW, BertForSequenceClassification\n",
    "from tqdm import tqdm, trange"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['KERAS_BACKEND'] = 'tensorflow'\n",
    "from keras.preprocessing.sequence import pad_sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ConvoKit\n",
    "As we alluded to in Week 6 with causal inference, [ConvoKit](https://convokit.cornell.edu/) is an exciting platform for conversational analysis developed by Jonathan Chang, Calem Chiam, and others, mostly at Cornell. Keep this in mind if you are interested in a final project with conversational data such as Twitter threads or movie scripts. They have an [interactive tutorial](https://colab.research.google.com/github/CornellNLP/Cornell-Conversational-Analysis-Toolkit/blob/master/examples/Introduction_to_ConvoKit.ipynb), which we include some examples from below. Most of the following text and code is authored by them.\n",
    "\n",
    "These ConvoKit corpora can be used for the next exercise in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EpQTyQjBUonG"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    import convokit\n",
    "except ModuleNotFoundError:\n",
    "    !pip install convokit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mMQXf1xHUonH"
   },
   "outputs": [],
   "source": [
    "# for pretty printing of cells within the Colab version of this notebook\n",
    "from IPython.display import HTML, display\n",
    "\n",
    "def set_css():\n",
    "  display(HTML('''\n",
    "  <style>\n",
    "    pre {\n",
    "        white-space: pre-wrap;\n",
    "    }\n",
    "  </style>\n",
    "  '''))\n",
    "get_ipython().events.register('pre_run_cell', set_css)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RLHbOOAzUonI",
    "outputId": "347f5c39-1dc4-411f-ab86-2649981d7cfc"
   },
   "outputs": [],
   "source": [
    "import convokit\n",
    "from convokit import Corpus, download"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "M5VIPx2oUonI"
   },
   "source": [
    "### Loading a Corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j_lhin9LUonJ"
   },
   "source": [
    "A Corpus represents a conversational dataset. We typically begin our analysis by loading a Corpus. A list of existing datasets already in ConvoKit format can be found [here](https://convokit.cornell.edu/documentation/datasets.html). \n",
    "\n",
    "A growing list of many other conversational datasets covering a variety of conversational settings are available in ConvoKit, such as face-to-face (e.g. the [*Intelligence Squared Debates corpus*](https://convokit.cornell.edu/documentation/iq2.html)), institutional (e.g. the [*Supreme Court Oral Arguments corpus*](https://convokit.cornell.edu/documentation/supreme.html)), fictional (e.g. the [*Cornell Movie Dialog Corpus*](https://convokit.cornell.edu/documentation/movie.html)), or online  (e.g. all talkpage conversations on [*Wikipedia Talk Pages*](https://convokit.cornell.edu/documentation/wiki.html) and a full dump of [*Reddit*](https://convokit.cornell.edu/documentation/subreddit.html)).\n",
    "\n",
    "For this tutorial, we will primarily be using the *r/Cornell* subreddit corpus to demo various ConvoKit functionality, and occasionally the [*Switchboard Dialog Act Corpus*](https://convokit.cornell.edu/documentation/switchboard.html) (a collection of anonymized five-minute telephone conversations) as a contrasting dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "488jR7pVUonJ",
    "outputId": "97013f50-3f8f-4453-e0bd-7d58550ff711"
   },
   "outputs": [],
   "source": [
    "corpus = Corpus(download('subreddit-Cornell'))\n",
    "\n",
    "# You can try a different corpus if you want.\n",
    "#corpus = Corpus(download('diplomacy-corpus'))\n",
    "#corpus = Corpus(download('switchboard-corpus'))\n",
    "#corpus = Corpus(download('reddit-corpus-small'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rnBisd3QUonL",
    "outputId": "8decac2b-80af-441a-c98f-1a833e715da4"
   },
   "outputs": [],
   "source": [
    "corpus.print_summary_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XTcJWCgzUonL"
   },
   "source": [
    "### Corpus components: Conversations, Utterances, Speakers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MnnTm_ALUonL"
   },
   "source": [
    "Every Corpus has three main components: [Conversations](https://convokit.cornell.edu/documentation/conversation.html), [Utterances](https://convokit.cornell.edu/documentation/utterance.html), and [Speakers](https://convokit.cornell.edu/documentation/speaker.html). Just as in real life, in ConvoKit, Conversations are some sequence of Utterances, where each Utterance is made by some Speaker. Let's look at an example of each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xW9eeQrcUonS",
    "outputId": "3488ceeb-da14-49e9-a25d-2f56091a6fa7"
   },
   "outputs": [],
   "source": [
    "# This is a Reddit thread\n",
    "corpus.random_conversation().meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TeR71mPgUonT",
    "outputId": "c2b1b327-90d7-4927-e8c6-7a4089c35802"
   },
   "outputs": [],
   "source": [
    "# This is a Reddit post or comment.\n",
    "corpus.random_utterance().meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LnQcLKBxUonS",
    "outputId": "d0b15e69-ea39-48c6-db15-63e2248dbe2a"
   },
   "outputs": [],
   "source": [
    "# The r/Cornell Corpus does not have speaker metadata.\n",
    "#corpus.random_speaker().meta\n",
    "\n",
    "#Speaker do have an 'id' which is their Reddit username, as seen here.\n",
    "corpus.random_speaker()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can iterate through these objects as we iterate lists or DataFrames in Python.\n",
    "for utt in corpus.iter_utterances():\n",
    "    print(utt.text)\n",
    "    break "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conversations, Utterances, and Speakers are each interesting, but the magic of conversational analysis is connecting them. For example, we can get all the Conversations in which a Speaker has participated and all the Utterances they have made. To make it more interesting, we can find a Speaker to study by navigating from a random Utterance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# consider this sequence of operations that highlight how to navigate between components\n",
    "utt = corpus.random_utterance()\n",
    "convo = utt.get_conversation() # get the Conversation the Utterance belongs to\n",
    "spkr = utt.speaker # get the Speaker who made the Utterance\n",
    "\n",
    "spkr_convos = list(spkr.iter_conversations())\n",
    "\n",
    "# Display up to 3 of their conversations.\n",
    "spkr_convos[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a more qualitative feel of the data, you can display a Conversation. For Reddit data, this is a single thread."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We truncate sentences at character 80 to avoid making this notebook too long!\n",
    "convo.print_conversation_structure(lambda utt: utt.text[:80] + \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a lot more to ConvoKit that we encourage you to explore, especially their [tutorial](https://colab.research.google.com/github/CornellNLP/Cornell-Conversational-Analysis-Toolkit/blob/master/examples/Introduction_to_ConvoKit.ipynb), but the ability to seamlessly navigate between the Utterances, Conversations, and Speakers of a Corpus is extremely valuable for social science."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"red\">*Exercise 1*</font>\n",
    "\n",
    "<font color=\"red\">Construct cells immediately below this that use ConvoKit to analyze a Corpus other than 'subreddit-Cornell', including at least one function you find in the package not used above. You can also generate a ConvoKit Corpus from your own dataset based on [their Corpus from .txt files tutorial](https://github.com/CornellNLP/Cornell-Conversational-Analysis-Toolkit/blob/master/examples/converting_movie_corpus.ipynb) or [their Corpus from pandas tutorial](https://github.com/CornellNLP/Cornell-Conversational-Analysis-Toolkit/blob/master/examples/corpus_from_pandas.ipynb), but that may be time-consuming for a weekly assignment. (It could be a great idea for your final project!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating networks of agents from corpora\n",
    "\n",
    "Now let's return to the Davies corpora (specifically, Soap Operas) to see how we can extract actors and build a network of their relationships in the texts.\n",
    "\n",
    "We'll use the `lucem_illud.loadDavies()` function to get the dataframe. Make sure to download `SOAP.zip` from DropBox, unzip, and edit the following line with the path to that file. This code may take some time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpora_address = \"C:/Downloads/SOAP\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soap_texts = lucem_illud.loadDavies(corpora_address, num_files=2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now use the source to see how the data is stored. Note that this is different from the movies corpus, and that we will need to use a different aggregating method to store the data. Each dataset would have a different approach, but they are all very similar, it depends on how the data is stored. Here multiple textids match multiple scripts, so our soap dataframe would be structured a little differently. \n",
    "\n",
    "You can see the first 20 lines of the source file here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "zfile = zipfile.ZipFile(corpora_address + \"/soap_sources.zip\")\n",
    "source = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for file in zfile.namelist():\n",
    "    with zfile.open(file) as f:\n",
    "        for line in f:\n",
    "            source.append(line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source[0:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soap_dict = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for soap in source[3:]:\n",
    "    try:\n",
    "        textID, year, show, url = soap.decode(\"utf-8\").split(\"\\t\")\n",
    "    except UnicodeDecodeError:\n",
    "        continue\n",
    "    if show.strip() not in soap_dict:\n",
    "        soap_dict[show.strip()] = []\n",
    "    if show.strip() in soap_dict:\n",
    "        try:\n",
    "            soap_dict[show.strip()].append(soap_texts[textID.strip()])\n",
    "        except KeyError:\n",
    "            continue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soap_dict.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soap_df = pd.DataFrame(columns=[\"Soap Name\", \"Tokenized Texts\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for soap in soap_dict:\n",
    "    # since there were multiple lists\n",
    "    print(soap)\n",
    "    full_script = []\n",
    "    for part in soap_dict[soap]:\n",
    "        full_script = full_script + part\n",
    "    soap_df.loc[i] = [soap, full_script]\n",
    "    i += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soap_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have each Soap, and each of the Tokenized texts. Let us see what kind of information we can get. These are American soap operas, and are likely to be cheesy and dramatic (an understatment). A fun start would be to make networks of each of the actors and actresses in these soaps. \n",
    "\n",
    "What would be a good way to create a network? Maybe everytime someone talks to someone we add one weight? But we wouldn't want to add weights whenever it's a different scene - or maybe we do? Let us look at the text and figure it out.\n",
    "\n",
    "Note that we didn't add the year here because it spans over multiple years. If we are doing different kinds of analysis we would want to a years column as well.\n",
    "\n",
    "In my dataframe, Days of Our Lives is the 4th corpora, and I conducted my basic analysis on that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dool = soap_df['Tokenized Texts'][3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "' '.join(dool[0:1500])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmmm... we can't do our normal text processing. But this provides us with an interesting oppurtunity: every '@!' is followed by some useeful information. Let us do a quick check of how many characters exist here, and how many times they speak."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "characters = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for token in dool:\n",
    "    if token[0] == '@':\n",
    "        # all characters or actions start with @, so we add that to character\n",
    "        if token[2:] not in characters:\n",
    "            characters[token[2:]] = 0\n",
    "        if token[2:] in characters:\n",
    "            characters[token[2:]] += 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(characters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow, that's a lot of characters: but we notice a '@!' between certain actions too, such as screaming and sobbing. Let us maybe only look for characters with a high number of appearances?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for character in characters:\n",
    "    if characters[character] > 2000:\n",
    "        print(character, characters[character])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check these folks out on the interwebz...a image of search of the name + \"days of our lives\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Image(filename='../data/dool/dool_john.png') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Image(filename='../data/dool/dool_brady.jpg') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Image(filename='../data/dool/dool_hope.jpeg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Image(filename='../data/dool/dool_philip.jpeg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Image(filename='../data/dool/dool_marlena.jpg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Image(filename='../data/dool/dool_kate.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Image(filename='../data/dool/dool_bo.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Image(filename='../data/dool/dool_chloe.jpg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Image(filename='../data/dool/dool_sami.jpg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Image(filename='../data/dool/dool_shawn.jpg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Image(filename='../data/dool/dool_belle.jpg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Image(filename='../data/dool/dool_lucas.jpg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Image(filename='../data/dool/dool_nicole.jpg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are definitely big, long-time players in the dramatic Days narrative. It would make sense to create a graph where each character who appears over 2000 times is a node, and each time they talk to each other, we add one to their weight. We should also store all the things these chracters say: that's useful information.\n",
    "\n",
    "So we now iterate through the tokens in a manner where we can capture this information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "actor_network = nx.Graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for character in characters:\n",
    "    if characters[character] > 2000:\n",
    "        actor_network.add_node(character, lines_spoken= characters[character], words=[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(actor_network.nodes.data())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "actor_network.nodes.data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "actor_network.nodes['Sami']['lines_spoken']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following lines of code creates the graph of actors and their relationships."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for token in dool:\n",
    "    i += 1\n",
    "    if i > len(dool):\n",
    "        break\n",
    "    if token[0] == \"@\":\n",
    "        if token[2:] in actor_network.nodes():\n",
    "            j = i\n",
    "            for token_ in dool[i:]:\n",
    "                if token_[0] == \"@\":\n",
    "                    # if both the characters exist in the graph, add a weight\n",
    "                    if token_[2:] != token[2:] and token_[2:] in actor_network.nodes():\n",
    "                        if (token[2:], token_[2:]) not in actor_network.edges():\n",
    "                            actor_network.add_edge(token[2:], token_[2:], weight=0)\n",
    "                        if (token[2:], token_[2:]) in actor_network.edges():\n",
    "                            actor_network.edges[(token[2:], token_[2:])]['weight'] += 1\n",
    "                    break\n",
    "                j += 1\n",
    "            # adding characters sentences\n",
    "            actor_network.nodes[token[2:]]['words'].append(dool[i:j])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.draw(actor_network, with_labels=True, font_weight='bold')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "L = []\n",
    "for node in actor_network.nodes():\n",
    "    l = []\n",
    "    for node_ in actor_network.nodes():\n",
    "        if node == node_:\n",
    "            l.append(0)\n",
    "        else:\n",
    "            l.append(actor_network.edges[(node, node_)]['weight'])\n",
    "    L.append(l)\n",
    "M_ = np.array(L)\n",
    "fig = plt.figure()\n",
    "div = pd.DataFrame(M_, columns = list(actor_network.nodes()), index = list(actor_network.nodes()))\n",
    "ax = sns.heatmap(div)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from networkx.algorithms.community import greedy_modularity_communities\n",
    "c = list(greedy_modularity_communities(actor_network))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding structure in networks\n",
    "\n",
    "We now have a lot of useful information: we have a graph of all the characters, with their relationships with other characters, as well as all the words they've said. We tried finding communities, but it seems like everyone is connected to everyone: each of them form their own 'community'. Seems like people talk to each other a bunch in soaps.\n",
    "\n",
    "This is however, not the best network to find any meaningful patterns, as we can see with everyone connected to everyone. But as we can see with our heatmap, not everyone talks to everyone an equal amount. How about we only keep our \"important\" ties, where people are talking to each other a lot?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "smaller_actor_network = nx.Graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for actor_1 in actor_network.nodes():\n",
    "    smaller_actor_network.add_node(actor_1, lines_spoken= actor_network.nodes[actor_1]['lines_spoken'], words=actor_network.nodes[actor_1]['words'])\n",
    "    for actor_2 in actor_network.nodes():\n",
    "        if actor_2!=actor_1 and actor_network.edges[(actor_1, actor_2)]['weight'] > 250:\n",
    "            smaller_actor_network.add_edge(actor_1, actor_2, weight=actor_network.edges[(actor_1, actor_2)]['weight'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.draw(smaller_actor_network, with_labels=True, font_weight='bold')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a lot more interesting: while the sets of characters overlap, there is still two distinct communities if you look at characters who regularly talk to each other!\n",
    "\n",
    "Let us see what our centrality measures look like, as well as communities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from networkx.algorithms.community import greedy_modularity_communities\n",
    "c = list(greedy_modularity_communities(smaller_actor_network))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dcentralities = nx.degree_centrality(smaller_actor_network)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dcentralities['John'], dcentralities['Philip']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our two different communities show up as detected by the networkx algorithm, and when we look at centralities, we can see that John is a lot more central than Philip."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us go back to our original graph, and see if the weight or number of similar appearences matches the text... how do we do this? Well, we already have the graph, and we also have information of who spoke to who. So we have our framework!\n",
    "\n",
    "This means we can explore ideas contained in two of the papers you will be reading: . “No country for old members: User lifecycle and linguistic change in online communities.”, and  “Fitting In or Standing Out? The Tradeoffs of Structural and Cultural Embeddedness”, both of which you can access on Canvas. \n",
    "\n",
    "Let us use a simplified version of the papers, and check if a higher number of conversations might lead to a higher similarity between the word distributions for two characters. We can use the same divergences we used in the last notebook. Do you think it will match with the number of times each character was associated with each other?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def kl_divergence(X, Y):\n",
    "    P = X.copy()\n",
    "    Q = Y.copy()\n",
    "    P.columns = ['P']\n",
    "    Q.columns = ['Q']\n",
    "    df = Q.join(P).fillna(0)\n",
    "    p = df.iloc[:,1]\n",
    "    q = df.iloc[:,0]\n",
    "    D_kl = scipy.stats.entropy(p, q)\n",
    "    return D_kl\n",
    "\n",
    "def chi2_divergence(X,Y):\n",
    "    P = X.copy()\n",
    "    Q = Y.copy()\n",
    "    P.columns = ['P']\n",
    "    Q.columns = ['Q']\n",
    "    df = Q.join(P).fillna(0)\n",
    "    p = df.iloc[:,1]\n",
    "    q = df.iloc[:,0]\n",
    "    return scipy.stats.chisquare(p, q).statistic\n",
    "\n",
    "def Divergence(corpus1, corpus2, difference=\"KL\"):\n",
    "    \"\"\"Difference parameter can equal KL, Chi2, or Wass\"\"\"\n",
    "    freqP = nltk.FreqDist(corpus1)\n",
    "    P = pd.DataFrame(list(freqP.values()), columns = ['frequency'], index = list(freqP.keys()))\n",
    "    freqQ = nltk.FreqDist(corpus2)\n",
    "    Q = pd.DataFrame(list(freqQ.values()), columns = ['frequency'], index = list(freqQ.keys()))\n",
    "    if difference == \"KL\":\n",
    "        return kl_divergence(P, Q)\n",
    "    elif difference == \"Chi2\":\n",
    "        return chi2_divergence(P, Q)\n",
    "    elif difference == \"KS\":\n",
    "        try:\n",
    "            return scipy.stats.ks_2samp(P['frequency'], Q['frequency']).statistic\n",
    "        except:\n",
    "            return scipy.stats.ks_2samp(P['frequency'], Q['frequency'])\n",
    "    elif difference == \"Wasserstein\":\n",
    "        try:\n",
    "            return scipy.stats.wasserstein_distance(P['frequency'], Q['frequency'], u_weights=None, v_weights=None).statistic\n",
    "        except:\n",
    "            return scipy.stats.wasserstein_distance(P['frequency'], Q['frequency'], u_weights=None, v_weights=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpora = []\n",
    "for character in actor_network.nodes():\n",
    "    character_words = []\n",
    "    for sentence in actor_network.nodes[character]['words']:\n",
    "        for word in sentence:\n",
    "            character_words.append(word)\n",
    "    corpora.append(lucem_illud.normalizeTokens(character_words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "L = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for p in corpora:\n",
    "    l = []\n",
    "    for q in corpora:\n",
    "        l.append(Divergence(p,q, difference='KS'))\n",
    "    L.append(l)\n",
    "M = np.array(L)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "div = pd.DataFrame(M, columns = list(actor_network.nodes()), index = list(actor_network.nodes()))\n",
    "ax = sns.heatmap(div)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# np.corrcoef(M_, M)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our two heatplots, we can attempt to do some rudimentary analysis. We can see from our previous plot that Shawn and Belle talk to each other a lot, so do Hope and Bo, and Nicole and Brady, and Lucas and Sami. Do they also talk *like* each other?\n",
    "\n",
    "Kind of, actually: all four of these pairs have a lower distance between them. Now I don't know anything about this particular soap... are these four pairs related? Are they in a relationship, either married or dating, or are they just really good friends?\n",
    "\n",
    "This lays out the frameworks which you can now use to explore your own networks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interactional influence\n",
    "\n",
    "Before we utilize transformers, let's see how to estimate the influence of one speaker on another in order to estimate a kind of interpersonal influence network based on a recent paper by Fangjian Guo, Charles Blundell, Hanna Wallach, and Katherine Heller entitled [\"The Bayesian Echo Chamber: Modeling Social Influence via Linguistic Accommodation\"](https://arxiv.org/pdf/1411.2674.pdf). This relies on a kind of point process called a Hawkes process that estimate the influence of one point on another. Specifically, what they estimate is the degree to which one actor to an interpersonal interaction engaged in \"accomodation\" behaviors relative to the other, generating a directed edge from the one to the other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First let's look at the output of their analysis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_name = '12-angry-men'   #example datasets: \"12-angry-men\" or \"USpresident\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_path = '../data/Bayesian-echo/results/{}/'.format(example_name)\n",
    "if not os.path.isdir(result_path):\n",
    "    raise ValueError('Invalid example selected, only \"12-angry-men\" or \"USpresident\" are avaliable')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_meta_info = pd.read_table(result_path + 'meta-info.txt',header=None)\n",
    "df_log_prob = pd.read_csv(result_path + \"SAMPLE-log_prior_and_log_likelihood.txt\",delim_whitespace=True) #log_prob samples\n",
    "df_influence = pd.read_csv(result_path + 'SAMPLE-influence.txt',delim_whitespace=True) # influence samples\n",
    "df_participants = pd.read_csv(result_path + 'cast.txt', delim_whitespace=True)\n",
    "person_id = pd.Series(df_participants['agent.num'].values-1,index=df_participants['agent.name']).to_dict()\n",
    "print()\n",
    "print ('Person : ID')\n",
    "person_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getDensity(df):\n",
    "    data = df#_log_prob['log.prior']\n",
    "    density = scipy.stats.gaussian_kde(data)\n",
    "    width = np.max(data) - np.min(data)\n",
    "    xs = np.linspace(np.min(data)-width/5, np.max(data)+width/5,600)\n",
    "    density.covariance_factor = lambda : .25\n",
    "    density._compute_covariance()\n",
    "    return xs, density(xs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot MCMC (Markov Monte Carlo) trace and the density of log-likelihoods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=[12,10])\n",
    "\n",
    "plt.subplot(4,2,1)\n",
    "plt.plot(df_log_prob['log.prior'])\n",
    "plt.xlabel('Iterations')\n",
    "plt.title('Trace of log.prior')\n",
    "\n",
    "plt.subplot(4,2,2)\n",
    "x,y = getDensity(df_log_prob['log.prior'])\n",
    "plt.plot(x,y)\n",
    "plt.xlabel('Iterations')\n",
    "plt.title('Density of log.prior')\n",
    "\n",
    "plt.subplot(4,2,3)\n",
    "plt.plot(df_log_prob['log.likelihood'])\n",
    "plt.title('Trace of log.likelihood')\n",
    "plt.xlabel('Iterations')\n",
    "plt.tight_layout()\n",
    "\n",
    "plt.subplot(4,2,4)\n",
    "x,y = getDensity(df_log_prob['log.likelihood'])\n",
    "plt.plot(x,y)\n",
    "plt.xlabel('Iterations')\n",
    "plt.title('Density of log.likelihood')\n",
    "\n",
    "plt.subplot(4,2,5)\n",
    "plt.plot(df_log_prob['log.likelihood.test.set'])\n",
    "plt.title('Trace of log.likelihood.test.set')\n",
    "plt.xlabel('Iterations')\n",
    "plt.tight_layout()\n",
    "\n",
    "plt.subplot(4,2,6)\n",
    "x,y = getDensity(df_log_prob['log.likelihood.test.set'])\n",
    "plt.plot(x,y)\n",
    "plt.xlabel('Iterations')\n",
    "plt.title('Density of log.likelihood.test.set')\n",
    "\n",
    "plt.subplot(4,2,7)\n",
    "plt.plot(df_log_prob['log.prior']+df_log_prob['log.likelihood'])\n",
    "plt.title('Trace of log.prob')\n",
    "plt.xlabel('Iterations')\n",
    "\n",
    "plt.subplot(4,2,8)\n",
    "x,y = getDensity(df_log_prob['log.prior']+df_log_prob['log.likelihood'])\n",
    "plt.plot(x,y)\n",
    "plt.xlabel('Iterations')\n",
    "plt.title('Density of log.prob')\n",
    "\n",
    "plt.tight_layout()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot the influence matrix between participants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = int(np.sqrt(len(df_influence.columns))) #number of participants\n",
    "id_person = {}\n",
    "for p in person_id:\n",
    "    id_person[person_id[p]]=p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getmatrix(stacked,A):\n",
    "    influence_matrix = [[0 for i in range(A)] for j in range(A)]\n",
    "    for row in stacked.iteritems():\n",
    "        from_ = int(row[0].split('.')[1])-1\n",
    "        to_ = int(row[0].split('.')[2])-1\n",
    "        value = float(row[1])\n",
    "        influence_matrix[from_][to_]=value\n",
    "    df_ = pd.DataFrame(influence_matrix) \n",
    "    \n",
    "    df_ =df_.rename(index = id_person)\n",
    "    df_ =df_.rename(columns = id_person)\n",
    "    return df_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacked = df_influence.mean(axis=0)\n",
    "df_mean = getmatrix(stacked,A)\n",
    "\n",
    "stacked = df_influence.std(axis=0)\n",
    "df_std = getmatrix(stacked,A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f, ax = plt.subplots(figsize=(9, 6))\n",
    "seaborn.heatmap(df_mean, annot=True,  linewidths=.5, ax=ax,cmap=\"YlGnBu\")\n",
    "print('MEAN of influence matrix (row=from, col=to)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f, ax = plt.subplots(figsize=(9, 6))\n",
    "seaborn.heatmap(df_std, annot=True,  linewidths=.5, ax=ax,cmap=\"YlGnBu\")\n",
    "print('SD of influence matrix (row=from, col=to)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Barplot of total influences sent/received"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sender_std = {} #sd of total influence sent\n",
    "reciever_std = {} #sd of total influence recieved\n",
    "for i in range(A):\n",
    "    reciever_std[id_person[i]] = df_influence[df_influence.columns[i::A]].sum(axis=1).std()\n",
    "    sender_std[id_person[i]] = df_influence[df_influence.columns[i*A:(i+1)*A:]].sum(axis=1).std()\n",
    "\n",
    "sent = df_mean.sum(axis=1) #mean of total influence sent\n",
    "recieved =df_mean.sum(axis=0) #mean of total influence recieved"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Total influence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (\"\\t\\tTotal linguistic influence sent/received \")\n",
    "ax.fig = plt.figure(figsize=[np.min([A,20]),6])\n",
    "\n",
    "plt.grid()\n",
    "wd=0.45\n",
    "ii=0\n",
    "for p in sender_std:\n",
    "    plt.bar(person_id[p],sent.loc[p],width=wd,color='red',alpha=0.6,label = \"Sent\" if ii == 0 else \"\")\n",
    "    plt.plot([person_id[p]-wd/4,person_id[p]+wd/4],[sent.loc[p]+sender_std[p],sent.loc[p]+sender_std[p]],color='k')\n",
    "    plt.plot([person_id[p]-wd/4,person_id[p]+wd/4],[sent.loc[p]-sender_std[p],sent.loc[p]-sender_std[p]],color='k')\n",
    "    plt.plot([person_id[p],person_id[p]],[sent.loc[p]-sender_std[p],sent.loc[p]+sender_std[p]],color='k')\n",
    "    ii+=1\n",
    "ii=0\n",
    "for p in reciever_std:\n",
    "    plt.bar(person_id[p]+wd,recieved.loc[p],width=wd,color='blue',alpha=0.4,label = \"Received\" if ii == 0 else \"\")\n",
    "    plt.plot([person_id[p]+wd-wd/4,person_id[p]+wd+wd/4],[recieved.loc[p]+reciever_std[p],recieved.loc[p]+reciever_std[p]],color='k')\n",
    "    plt.plot([person_id[p]+wd-wd/4,person_id[p]+wd+wd/4],[recieved.loc[p]-reciever_std[p],recieved.loc[p]-reciever_std[p]],color='k')\n",
    "    plt.plot([person_id[p]+wd,person_id[p]+wd],[recieved.loc[p]-reciever_std[p],recieved.loc[p]+reciever_std[p]],color='k')\n",
    "    ii+=1\n",
    "plt.legend(loc='center left', bbox_to_anchor=(1, 0.7))\n",
    "plt.xticks([i+0.25 for i in range(A)],list(zip(*sorted(id_person.items())))[1])\n",
    "plt.ylabel('value')\n",
    "plt.xlabel('speaker',fontsize=14)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize Influence Network!\n",
    "\n",
    "You can visualize any of the influence matrices above:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using networkx:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def drawNetwork(df,title):\n",
    "    fig = plt.figure(figsize=[8,8])\n",
    "    G = nx.DiGraph()\n",
    "    for from_ in df.index:\n",
    "        for to_ in df.columns:\n",
    "            G.add_edge(from_,to_,weight = df.loc[from_][to_])\n",
    "            \n",
    "    pos = nx.spring_layout(G,k=0.55,iterations=20)\n",
    "    edges,weights = zip(*nx.get_edge_attributes(G,'weight').items())\n",
    "    weights = np.array(weights)\n",
    "    #weights = weights*weights\n",
    "    weights = 6*weights/np.max(weights)\n",
    "    print(title)\n",
    "    \n",
    "    edge_colors=20*(weights/np.max(weights))\n",
    "    edge_colors = edge_colors.astype(int)\n",
    "#     nx.draw_networkx_nodes(G,pos,node_size=1200,alpha=0.7,node_color='#99cef7')\n",
    "#     nx.draw_networkx_edges(G,pos,edge_color=edge_colors)\n",
    "#     nx.draw_networkx_labels(G,pos,font_weight='bold')\n",
    "    nx.draw(G,pos,with_labels=True, font_weight='bold',width=weights,\\\n",
    "            edge_color=255-edge_colors,node_color='#99cef7',node_size=1200,\\\n",
    "            alpha=0.75,arrows=True,arrowsize=20)\n",
    "    return edge_colors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get quantile influence matrices for 25%, 50%, 75% quantile\n",
    "stacked = df_influence.quantile(0.25)\n",
    "df_q25 = getmatrix(stacked,A)\n",
    "\n",
    "stacked = df_influence.quantile(0.5)\n",
    "df_q50 = getmatrix(stacked,A)\n",
    "\n",
    "stacked = df_influence.quantile(0.75)\n",
    "df_q75 = getmatrix(stacked,A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "G_mean = drawNetwork(df_mean,'Mean Influence Network')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "G_q25 = drawNetwork(df_q25,'25 Quantile Influence Network')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "G_q75 = drawNetwork(df_q75,'75 Quantile Influence Network')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fakeEnglish(length):\n",
    "    listd=['a','b','c','d','e','f','g','s','h','i','j','k','l']\n",
    "    return ''.join(np.random.choice(listd,length))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your own dataset should contains 4 columns (with the same column names) as the artificial one below:\n",
    "\n",
    "- name: name of the participant\n",
    "- tokens: a list of tokens in one utterance\n",
    "- start: starting time of utterance (unit doesn't matter, can be 'seconds','minutes','hours'...)\n",
    "- end: ending time of utterance (same unit as start)\n",
    "\n",
    "There is no need to sort data for the moment.\n",
    "\n",
    "Below, we generate a fake collection of data from \"Obama\", \"Trump\", \"Clinton\"...and other recent presidents. You can either create your own simulation OR (better), add real interactional data from a online chat forum, comment chain, or transcribed from a conversation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "script= []\n",
    "language = 'eng' #parameter, no need to tune if using English, accept:{'eng','chinese'}\n",
    "role = 'Adult' #parameter, no need to tune \n",
    "\n",
    "for i in range(290):\n",
    "    dt = []\n",
    "    dt.append(np.random.choice(['Obama','Trump','Clinton','Bush','Reagan','Carter','Ford','Nixon','Kennedy','Roosevelt']))\n",
    "    faketokens = [fakeEnglish(length = 4) for j in range(30)]\n",
    "    dt.append(faketokens) #fake utterance\n",
    "    dt.append(i*2+np.random.random()) # start time\n",
    "    dt.append(i*2+1+np.random.random()) # end time\n",
    "    script.append(dt)\n",
    "\n",
    "df_transcript = pd.DataFrame(script,columns=['name','tokens','start','end']) #\"start\", \"end\" are timestamps of utterances, units don't matter\n",
    "df_transcript[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transform data into TalkbankXML format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_fname = 'USpresident.xml'  #should be .xml\n",
    "language = 'eng' \n",
    "#language = 'chinese'\n",
    "lucem_illud.make_TalkbankXML(df_transcript, output_fname, language = language )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run Bayesian Echo Chamber to get estimation.\n",
    "\n",
    "- It may take a couple of hours. ( About 4-5 hours if Vocab_size=600 and sampling_time =2000)\n",
    "- Larger \"Vocab_size\" (see below) will cost more time\n",
    "- Larger \"sampling_time\" will also consume more time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Vocab_size = 90 # up to Vocab_size most frequent words will be considered, it should be smaller than the total vocab\n",
    "sampling_time = 1500  #The times of Gibbs sampling sweeps  (500 burn-in not included)\n",
    "lucem_illud.bec_run(output_fname, Vocab_size, language, sampling_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"red\">*Exercise 2*</font>\n",
    "\n",
    "<font color=\"red\">Construct cells immediately below this that perform a similar social similarity or influence analysis on a dataset relevant to your final project (__or one from ConvoKit__). Create relationships between actors in a network based on your dataset (e.g., person to person or document to document), and perform analyses that interrogate the structure of their interactions, similarity, and/or influence on one another. (For example, if relevant to your final project, you could explore different soap operas, counting how many times a character may have used the word love in conversation with another character, and identify if characters in love speak like each other. Or do opposites attract?) What does that analysis and its output reveal about the relative influence of each actor on others? What does it reveal about the social game being played?\n",
    "\n",
    "<font color=\"red\">Stretch 1:\n",
    "Render the social network with weights (e.g., based on the number of scenes in which actors appear together), then calculate the most central actors in the show.Realtime output can be viewed in shell.\n",
    "\n",
    "<font color=\"red\">Stretch 2:\n",
    "Implement more complex measures of similarity based on the papers you have read."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Generation using GPT-2 and BERT\n",
    "\n",
    "We can make use of the transformers we learned about last week to do text generation, where the model takes one or multiple places in a conversation. While some may regard it as a parlour trick due to unpredictability, recent dramatic improvements in text generation suggest that these kind of models can find themselves being used in more serious social scientific applications, such as in survey design and construction, idiomatic translation, and the normalization of phrase and sentence meanings.\n",
    "\n",
    "These models can be quite impressive, even uncanny in how human like they sound. Check out this [cool website](https://transformer.huggingface.co), which allows you to write with a transformer. The website is built by the folks who wrote the package we are using. The code underneath the website can be found in their examples: [run_generation.py](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py).\n",
    "\n",
    "We will be using the built in generate function, but the example file has more detailed code which allows you to set the seed differently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoModelWithLMHead, AutoTokenizer\n",
    "\n",
    "tokenizer_gpt = AutoTokenizer.from_pretrained(\"gpt2\")\n",
    "model_gpt = AutoModelWithLMHead.from_pretrained(\"gpt2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sequence = \"Nothing that we like to do more than analyse data all day long and\"\n",
    "\n",
    "input = tokenizer_gpt.encode(sequence, return_tensors=\"pt\")\n",
    "generated = model_gpt.generate(input, max_length=50)\n",
    "\n",
    "resulting_string = tokenizer_gpt.decode(generated.tolist()[0])\n",
    "print(resulting_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow. A little creepy, and as we can see, far from perfect: GPT doesn't alwats work out flawlessly, but it sometimes can, and we will try and see if fine-tuning helps. We are going to tune the model on a complete dataset of Trump tweets, as they have a set of distinctive, highly identifiable qualities.\n",
    "\n",
    "### Creating a domain-specific language model\n",
    "\n",
    "One of the most exciting things about BERT and GPT is being able to retune them the way we want to. We will be training models to perform two tasks - one is to create a BERT with an \"accent\", by traning a model with english news data from the UK, from the US, and from India. We will also train a language generation model with a bunch of Trump tweets. \n",
    "\n",
    "We can train models specifically over a certain domain to make its language generation similar to that domain. \n",
    "`run_language modeling.py` followed by `run_generation.py`. I've downloaded these files and added them to this directory so we can run them through the notebook. You are encouraged to look at these files to get a rough idea of what is going on.\n",
    "\n",
    "### Loading Data \n",
    "\n",
    "We want to now get our Trump tweets and our English news datasets ready. The data the scripts expect is just a text file with relevant data. We load the Trump tweets and then write them to disk as train and test files with only data. I leave the original dataframes in case you would like to use it for your own purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for file in os.listdir(\"../data/trump_tweets\"):\n",
    "    dfs.append(pd.read_json(\"../data/trump_tweets/\" + file))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat(dfs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "train_text, test_text = train_test_split(df['text'], test_size=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_text.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_text.to_frame().to_csv(r'train_text_trump', header=None, index=None, sep=' ', mode='a')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_text.to_frame().to_csv(r'test_text_trump', header=None, index=None, sep=' ', mode='a')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I now used the Google Colab GPUs to train the Trump tweet models. We'll be doing the same for our blog posts too.\n",
    "\n",
    "### GloWBe dataset\n",
    "\n",
    "We'll now load up the GloWbe (Corpus of Global Web-Based English) dataset which have different texts from different countries. We'll try and draw out texts from only the US, UK and India. We'll then save these to disk. Note that this is a Davies Corpora dataset: the full download can be done with the Dropbox link I sent in an announcement a few weeks ago. The whole download is about 3.5 GB but we only need two files, which are anout 250 MB each. The other files might be useful for your research purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "address = \"C:/Downloads/GloWbE\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# these are the exact name of the files\n",
    "us = \"/text_us_blog_jfy.zip\"\n",
    "gb = \"/text_gb_blog_akq.zip\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "us_texts = lucem_illud.loadDavies(address, corpus_style=\"us_blog\", num_files=5000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb_texts = lucem_illud.loadDavies(address, corpus_style=\"gb_blog\", num_files=5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have a dictionary with document ids mapping to text. Since we don't need any information but the text, we can just save these to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "' '.join(list(us_texts.values())[10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def dict_to_texts(texts, file_name):\n",
    "    text = []\n",
    "    for doc in list(texts.values()):\n",
    "        text.append(' '.join(doc).replace(\"< h >\", \"\").replace(\"< p >\", \"\"))\n",
    "    train_text, test_text = train_test_split(text, test_size=0.2)\n",
    "    with open(file_name + \"_train\", 'w') as f:\n",
    "        for item in train_text:\n",
    "            f.write(\"%s\\n\" % item)\n",
    "    \n",
    "    with open(file_name + \"_test\", 'w') as f:\n",
    "        for item in test_text:\n",
    "            f.write(\"%s\\n\" % item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dict_to_texts(us_texts, \"us_blog\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dict_to_texts(gb_texts, \"gb_blog\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have the training and testing files for both US and GB blogs in English. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(WARNING - SHIFT TO GOOGLE COLAB OR GPU ENABLED MACHINE)\n",
    "\n",
    "\n",
    "### Running Scripts\n",
    "\n",
    "We use the scripts to do language modeling and text generation. The following cells run the code as if you would have run it in a terminal.\n",
    "\n",
    "#### Trump GPT-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You might have issues with the memory of your GPU in the following code.\n",
    "# The default Google Colab GPU should work with batch size of 2,\n",
    "# but if you get a \"CUDA out of memory\" error, you can reduce it to 1.\n",
    "\n",
    "# !python run_language_modeling_gpt.py --per_gpu_train_batch_size=1 --output_dir=output_gpt_trump --model_type=gpt2 --model_name_or_path=gpt2 --do_train --train_data_file=train_text_trump --do_eval --eval_data_file=test_text_trump\n",
    "!python run_language_modeling_gpt.py --per_gpu_train_batch_size=2 --output_dir=output_gpt_trump --model_type=gpt2 --model_name_or_path=gpt2 --do_train --train_data_file=train_text_trump --do_eval --eval_data_file=test_text_trump\n",
    "# !python run_language_modeling_gpt.py --output_dir=output_gpt_trump --model_type=gpt2 --model_name_or_path=gpt2 --do_train --train_data_file=train_text_trump --do_eval --eval_data_file=test_text_trump\n",
    "\n",
    "# If the GPU memory error still occurs even with a batch size of 1,\n",
    "# you might need to run the code on RCC Midway\n",
    "# (See https://github.com/UChicago-Computational-Content-Analysis/Frequently-Asked-Questions/issues/18)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### RoBERTa US"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python run_language_modeling_roberta.py --per_gpu_train_batch_size=1 --output_dir=output_roberta_us --model_type=roberta --model_name_or_path=roberta-base --do_train --train_data_file=us_blog_train --do_eval --eval_data_file=us_blog_test --mlm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### RoBERTa UK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python run_language_modeling_roberta.py --output_dir=output_roberta_gb --model_type=roberta --model_name_or_path=roberta-base --do_train --train_data_file=gb_blog_train --do_eval --eval_data_file=gb_blog_test --mlm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(If you left to use a GPU machine, COME BACK TO THIS NOTEBOOK to load and work with your trained model.)\n",
    "\n",
    "### Loading and using models\n",
    "\n",
    "Let us now load the four models we have and see how we can use them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now - let us see what our Trump Tweet Bot looks like!\n",
    "You can generate text via command line using the command below. You can also load a model once it is saved - I trained my model using Google Colab, downloaded the model, and am loading it again via the command below. Note that you have to download all the files in your folder of the fine-tuned model to use the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !python run_generation.py --model_type=gpt2 --model_name_or_path=output_gpt_trump"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer_trump = AutoTokenizer.from_pretrained(\"output_gpt_trump\")\n",
    "# If this line does not work, try tokenizer_trump = AutoTokenizer.from_pretrained(\"gpt2\")\n",
    "\n",
    "model_trump = AutoModelWithLMHead.from_pretrained(\"output_gpt_trump\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sequence = \"Obama is going to\"\n",
    "\n",
    "input = tokenizer_trump.encode(sequence, return_tensors=\"pt\")\n",
    "generated = model_trump.generate(input, max_length=50, bos_token_id=1, pad_token_id=1, eos_token_ids=1)\n",
    "\n",
    "resulting_string = tokenizer_trump.decode(generated.tolist()[0])\n",
    "print(resulting_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow - our Trump bot is nasty, so we know our model trained well. What happens if we try the same sentence for our non-fine tuned model?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sequence = \"Obama is going to\"\n",
    "\n",
    "input = tokenizer_gpt.encode(sequence, return_tensors=\"pt\")\n",
    "generated = model_gpt.generate(input, max_length=50, bos_token_id=1, pad_token_id=1, eos_token_ids=1)\n",
    "\n",
    "resulting_string = tokenizer_gpt.decode(generated.tolist()[0])\n",
    "print(resulting_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Quite the contrast."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"red\">*Exercise 3*</font>\n",
    "\n",
    "<font color=\"red\">Construct cells immediately below this that generate a BERT-powered chatbot tuned on text related to your final project. What is interesting about this model, and how to does it compare to an untrained model? What does it reveal about the social game involved with your dataset?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Contextual models can also help us visualize how words in a sentence or different or similar to each other. We will try to construct sentences where words might mean different things in different countries - in the US, people might eat chips with salsa, but in the UK, chips are what Americans call french fries, and might eat it fried fish instead. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import RobertaConfig, RobertaModel, RobertaTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "roberta_us_model_embedding = RobertaModel.from_pretrained('roberta_us')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "roberta_us_tokenizer = RobertaTokenizer.from_pretrained('roberta_us')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"Do you have your chips with fish or with salsa?\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text1 = \"He went out in just his undershirt and pants.\" #pants are underwear in Britain; maybe closer to an undershirt\n",
    "text2 = \"His braces completed the outfit.\" #braces are suspenders (in Britain); maybe closer to an outfit\n",
    "text3 = \"Does your pencil have a rubber on it?\" #rubber is an eraser in Britain); maybe closer to a pencil\n",
    "text4 = \"Was the bog closer to the forest or the house?\" #bog is a toilen in Britain); maybe closer to a house\n",
    "text5 = \"Are you taking the trolley or the train to the grocery market\" #trolley is a food carriage; possibly closer to a market"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def word_vector(text, word_id, model, tokenizer):\n",
    "    marked_text = \"[CLS] \" + text + \" [SEP]\"\n",
    "    tokenized_text = tokenizer.tokenize(marked_text)\n",
    "    indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)\n",
    "    tokens_tensor = torch.tensor([indexed_tokens])\n",
    "    word_embeddings = model(tokens_tensor)[0]\n",
    "    sentence_embeddings = model(tokens_tensor)[1]\n",
    "    vector = word_embeddings[0][word_id].detach().numpy()\n",
    "    return vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def visualise_diffs(text, model, tokenizer):\n",
    "    word_vecs = []\n",
    "    for i in range(0, len(text.split())):\n",
    "        word_vecs.append(word_vector(text, i, model, tokenizer))\n",
    "    L = []\n",
    "    for p in word_vecs:\n",
    "        l = []\n",
    "        for q in word_vecs:\n",
    "            l.append(1 - cosine(p, q))\n",
    "        L.append(l)\n",
    "    M = np.array(L)\n",
    "    fig = plt.figure()\n",
    "    div = pd.DataFrame(M, columns = list(text.split()), index = list(text.split()))\n",
    "    ax = sns.heatmap(div)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visualise_diffs(text, roberta_us_model_embedding, roberta_us_tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "roberta_gb_model_embedding = RobertaModel.from_pretrained('roberta_gb')\n",
    "roberta_gb_tokenizer = RobertaTokenizer.from_pretrained('roberta_gb')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visualise_diffs(text, roberta_gb_model_embedding, roberta_gb_tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"red\">*Exercise 4*</font>\n",
    "\n",
    "<font color=\"red\">What's a research question relevant to your final project that would benefit from a generative or conversational model like GPT-2 or GPT-3? What task would you have it perform, and how would you evaluate the ouput?\n",
    "    \n",
    "<font color=\"red\">_Stretch:_ Do it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optional: AI Dungeon\n",
    "A fun way to explore the power of conversation and text generation is [AI Dungeon](https://play.aidungeon.io/main/newGame). If you select _Quick Start_, you can start a text adventure similar to the popular role-playing game Dungeons and Dragons, or the \"choose your own adventure\" books some of us boomers read as kids. AI Dungeon used to use GPT-2 but now uses GPT-3 to generate text in response to your actions. For example, you may begin with the prompt:\n",
    "\n",
    "```\n",
    "You are Jimmy, a wizard living in the kingdom of Larion. You have a staff and a spellbook. You finish your long journey and finally arrive at the ruin you've been looking for. You have come here searching for a mystical spellbook of great power called the book of essence. You look around and see a beautiful woman with raven hair and dark, sparkling eyes. She approaches you and asks you what you seek.\n",
    "\n",
    "You are now before the one known as Raven.\n",
    "```\n",
    "\n",
    "You can respond with any text. I type, `I ask Raven, \"What is the meaning of life?\"`, to which AI Dungeon says:\n",
    "\n",
    "```\n",
    "Raven ponders the question for a moment, and then she answers. Raven says, \"It is the pursuit of happiness.\" What is the pursuit of happiness?\n",
    "```\n",
    "\n",
    "I type, `I respond, \"Happiness to me would be unearthing the treasures of the Kilarean Empire.\"`:\n",
    "\n",
    "```\n",
    "Raven looks at me and says, \"Come with me and live out your one true life.\"\n",
    "\n",
    "You go with Raven through many passages and up and down elevators.\n",
    "```\n",
    "\n",
    "Raven might be helping me unearth the treasures! Not bad!\n",
    "\n",
    "There are a lot of tools like AI Dungeon being developed now with state-of-the-art language models. Now that you have some understanding of deep language models, you can better appreciate them—and perhaps design some of your own!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}