convokit/expected_context_framework/demos/switchboard_exploration_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Applying the Expected Context Framework to the Switchboard Corpus\n",
    "\n",
    "### Using `ExpectedContextModelTransformer`\n",
    "\n",
    "This notebook demonstrates how our implementation of the Expected Context Framework can be applied to the Switchboard dataset. See [this dissertation](https://tisjune.github.io/research/dissertation) for more details about the framework, and more comments on the below analyses.\n",
    "\n",
    "This notebook will show how to apply two related instances of `ExpectedContextModelTransformer`. For a version of this demo that uses `DualContextWrapper`, a wrapper transformer around these two instances, see [this notebook](https://github.com/CornellNLP/ConvoKit/blob/ecf/convokit/expected_context_framework/demos/switchboard_exploration_dual_demo.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import math\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Loading and preprocessing the dataset\n",
    "\n",
    "For this demo, we'll use the Switchboard corpus---a collection of telephone conversations which have been annotated with various dialog acts. More information on the dataset, as it exists in ConvoKit format, can be found [here](https://convokit.cornell.edu/documentation/switchboard.html); the original data is described [here](https://web.stanford.edu/~jurafsky/ws97/CL-dialog.pdf).\n",
    "\n",
    "We will actually use a preprocessed version of the Switchboard corpus, which we can access below. Since Switchboard consists of transcribed telephone conversations, there are many disfluencies and backchannels, that make utterances messier, and that make it hard to identify what counts as an actual turn. In the version of the corpus we consider, for the purpose of demonstration, we remove the disfluencies and backchannels (acknowledging that we're discarding important parts of the conversations). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from convokit import Corpus\n",
    "from convokit import download"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# OPTION 1: DOWNLOAD CORPUS \n",
    "# UNCOMMENT THESE LINES TO DOWNLOAD CORPUS\n",
    "# DATA_DIR = '<YOUR DIRECTORY>'\n",
    "# SW_CORPUS_PATH = download('switchboard-processed-corpus', data_dir=DATA_DIR)\n",
    "\n",
    "# OPTION 2: READ PREVIOUSLY-DOWNLOADED CORPUS FROM DISK\n",
    "# UNCOMMENT THIS LINE AND REPLACE WITH THE DIRECTORY WHERE THE TENNIS-CORPUS IS LOCATED\n",
    "# SW_CORPUS_PATH = '<YOUR DIRECTORY>'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sw_corpus = Corpus(SW_CORPUS_PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Speakers: 440\n",
      "Number of Utterances: 44402\n",
      "Number of Conversations: 1155\n"
     ]
    }
   ],
   "source": [
    "sw_corpus.print_summary_stats()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "utt_eg_id = '3496-79'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as input, we use a preprocessed version of the utterance that only contains alphabetical words, found in the `alpha_text` metadata field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'How old were you when you left'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sw_corpus.get_utterance(utt_eg_id).meta['alpha_text']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to avoid capturing topic-specific information, we restrict our analyses to a vocabulary of unigrams that occurs across many topics, and across many conversations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "topic_counts = defaultdict(set)\n",
    "for ut in sw_corpus.iter_utterances():\n",
    "    topic = sw_corpus.get_conversation(ut.conversation_id).meta['topic']\n",
    "    for x in set(ut.meta['alpha_text'].lower().split()):\n",
    "        topic_counts[x].add(topic)\n",
    "topic_counts = {x: len(y) for x, y in topic_counts.items()}\n",
    "\n",
    "word_convo_counts = defaultdict(set)\n",
    "for ut in sw_corpus.iter_utterances():\n",
    "    for x in set(ut.meta['alpha_text'].lower().split()):\n",
    "        word_convo_counts[x].add(ut.conversation_id)\n",
    "word_convo_counts = {x:  len(y) for x, y in word_convo_counts.items()}\n",
    "\n",
    "min_topic_words = set(x for x,y in topic_counts.items() if y >= 33)\n",
    "min_convo_words = set(x for x,y in word_convo_counts.items() if y >= 200)\n",
    "vocab = sorted(min_topic_words.intersection(min_convo_words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "381"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from convokit.expected_context_framework import ColNormedTfidfTransformer, ExpectedContextModelTransformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Applying the Expected Context Framework\n",
    "\n",
    "To apply the Expected Context Framework, we start by converting the input utterance text to an input vector representation. Here, we represent utterances in a term-document matrix that's _normalized by columns_ (empirically, we found that this ensures that the representations derived by the framework aren't skewed by the relative frequency of utterances). We use `ColNormedTfidfTransformer` transformer to do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tfidf_obj = ColNormedTfidfTransformer(input_field='alpha_text', output_field='col_normed_tfidf', binary=True, vocabulary=vocab)\n",
    "_ = tfidf_obj.fit(sw_corpus)\n",
    "_ = tfidf_obj.transform(sw_corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now use the Expected Context Framework. In short, the framework derives vector representations, and other characterizations, of terms and utterances that are based on their _expected conversational context_---i.e., the replies we expect will follow a term or utterance, or the preceding utterances that we expect the term/utterance will reply to. \n",
    "\n",
    "We start by applying the framework to derive characterizations based on the _forwards_ context, i.e., the expected replies. Here, we initialize a framework transformer object `ec_fw`.\n",
    "We initialize it with the following arguments:\n",
    "* `next_id` is a field in each utterance's metadata that indicates the ID of its reply, i.e., the context that we will use.\n",
    "* `output_prefix` determines the names of the matrices and metadata fields that the framework object outputs, when we later call `ec_fw.transform(corpus)` (see below)\n",
    "* `vect_field` and `context_vect_field` respectively denote the input vector representations of utterances and context utterances that `ec_fw` will work with. Here, we'll use the same tf-idf representations that we just computed above.\n",
    "* `n_svd_dims` denotes the dimensionality of the vector representations that `ec_fw` will output. This is something that you can play around with---for this dataset, we found that more dimensions resulted in messier output, and a coarser, lower-dimensional representation was slightly more interpretable. (Technical note: technically, `ec_fw` produces vector representations of dimension `n_svd_dims`-1, since by default, it removes the first latent dimension, which we find tends to strongly reflect term frequency.)\n",
    "* `n_clusters` denotes the number of utterance types that `ec_fw` will infer, given the representations it computes. Note that this is an interpretative step: looking at clusters of utterances helps us get a sense of what information the representations are capturing; this value does not actually impact the representations and other characterizations we derive.\n",
    "* `random_state` and `cluster_random_state` are fixed for this demo, so we produce deterministic output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ec_fw = ExpectedContextModelTransformer(context_field='next_id', output_prefix='fw', \n",
    "                                    vect_field='col_normed_tfidf', context_vect_field='col_normed_tfidf', \n",
    "                                      n_svd_dims=15, n_clusters=2,\n",
    "                                     random_state=1000, cluster_random_state=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll fit the `ec_fw` transformer on the subset of utterances and replies that have at least 5 unigrams from our vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ec_fw.fit(sw_corpus, selector=lambda x: x.meta.get('col_normed_tfidf__n_feats',0)>=5, \n",
    "            context_selector=lambda x: x.meta.get('col_normed_tfidf__n_feats',0)>= 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we apply the framework to derive characterizations based on the _backwards_ context, i.e., the expected predecessors. As such, we set `context_field='reply_to'`, meaning the `ec_bk` transformer will use derive characterizations based on predecessors.\n",
    "\n",
    "Since we want the representations derived in the backwards direction to be comparable to that derived in the forwards direction, we initialize the backwards framework object, `ec_bk`, with the forwards one, via the argument `model=ec_fw`. (Under the hood, `ec_bk` is initialized with the latent context vectors that `ec_fw` has derived.) For a demonstration of `DualContextWrapper`, a wrapper that handles both expected context models we've initialized, see [this notebook](https://github.com/CornellNLP/ConvoKit/blob/ecf/convokit/expected_context_framework/demos/switchboard_exploration_dual_demo.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ec_bk = ExpectedContextModelTransformer(context_field='reply_to', output_prefix='bk', \n",
    "                                    vect_field='col_normed_tfidf', context_vect_field='col_normed_tfidf', \n",
    "                                      n_svd_dims=15, n_clusters=2,\n",
    "                                     random_state=1000, cluster_random_state=1000,\n",
    "                                        model=ec_fw)\n",
    "ec_bk.fit(sw_corpus, selector=lambda x: x.meta.get('col_normed_tfidf__n_feats',0)>=5, \n",
    "            context_selector=lambda x: x.meta.get('col_normed_tfidf__n_feats',0)>= 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpreting derived representations\n",
    "\n",
    "Before applying the two transformers, `ec_fw` and `ec_bk` to transform the corpus, we can examine the representations and characterizations it's derived over the training data (note that in this case, the training data is also the corpus that we analyze, but this needn't be the case in general---see [this demo](https://github.com/CornellNLP/ConvoKit/blob/master/convokit/expected_context_framework/demos/wiki_awry_demo.ipynb) for an example)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import paired_distances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, to interpret the representations derived by each model, we can inspect the clusters of representations that we've inferred, for both the forwards and backwards direction. The following function calls print out representative terms and utterances, as well as context terms and utterances, per cluster (next two cells; note that the output is quite long). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CLUSTER 0 0\n",
      "---\n",
      "terms\n",
      "         cluster_dist\n",
      "index                \n",
      "to           0.339085\n",
      "that         0.340681\n",
      "is           0.398213\n",
      "not          0.407413\n",
      "know         0.413964\n",
      "for          0.421002\n",
      "with         0.429641\n",
      "about        0.429842\n",
      "because      0.444867\n",
      "me           0.447905\n",
      "\n",
      "context terms\n",
      "       cluster_dist\n",
      "index              \n",
      "that       0.464557\n",
      "true       0.517588\n",
      "do         0.558011\n",
      "think      0.579059\n",
      "know       0.750303\n",
      "sure       0.751195\n",
      "what       0.762769\n",
      "no         0.825287\n",
      "right      0.846140\n",
      "how        0.870710\n",
      "\n",
      "\n",
      "utterances\n",
      "> 2303-21 0.109 Yeah , that 's , that 's possible . I still think that a lot of those people are the ones who really think that their votes do n't make a difference , though , as well . I think it 's those same people who do n't know any better about how we vote , are , are , are a lot of the people who think that well , look at me , I 'm just a little nobody . My vote 's not going to count anyway . You know , and I think that 's probably a portion of the population that massively under represent- , - I , I , I would guess that that portion of the population is massively under represented .\n",
      "> 2041-96 0.113 Yeah , yeah . Exactly . And , part of it is California , you know , in , back in the sixties , had a lot of alternative movements and some of them fizzled out and some of them were disastrous and others of them , um , had an impact on the society around here . And one of the ones that had an impact was , uh , people becoming interested in alternate practices , ( ( I 'm not sure ) ) if it was a meditation practice , or if it was , you know , which is similar to a stress management practice or alternates to , uh , A M A approved medicine . Uh , you have , you know , major , um , acupuncture schools and things out here . And , and you could have them around long enough and more and more people start believing them or wondering how to combine them with other things , and , before long , you , you get this , this whole kind of Gestalt , this whole package of , of health , of health care and options -- -- and , uh , and exercise is , is strongly considered one of them . The irony is , is its\n",
      "> 3924-31 0.113 Well , and one of the things I 'd , - and I 'm with you that I do n't know is , is that really the only way it 's being transmitted or was that the easier way to tell us and we 're going to find out more .\n",
      "> 3343-53 0.119 Ye- , I think technology is better , I 'm , I 'm not sure . I think -- -- you 've got a good point with the plastic and that . I do n't think necessarily that , things are being made better -- -- uh , you know , I , I , - everything is so automated , uh , and things can be made without , I think -- -- a human hand ever touching it , you know -- -- an awful lot of things , and I think that certainly makes a difference from , back from the time when somebody handcrafted something and , you know , had some pride in it .\n",
      "> 2780-144 0.120 I think I I Yeah I I d- I guess the idea you know they always keep saying like the framers . - Um . there uh one of the Republican the appointees for the jud- the judicial bench - and they always talk about ( ( ) ) they feel that this person believes in the framing of the constitution what the original framers the ideals the original framers set down . # And Um # I believe that the idea of burning the flag is is in - my understanding of how it was framed is that you this coun- the thing about this country is that you can disagree with its government and you can display Uh , I Yeah , I g- - and and so that as a freedom of expression and speech and whatever I I think that that 's viable . Um as long as it 's your flag and and you 've made your your point . I think . Yeah Because , realistically , like for you for you you do n't think it 's right And I would never do it . I I love this country too much , and that symbol means a lot to me . I guess it 's just one of those things where if t\n",
      "> 3597-15 0.124 I do too . I think part of that , - I do n't know , I think the role of women changing has been good in that women are feeling like , in terms of their self confid- , confidence , their self worth , you know , I can do something , I can be somebody , you know , if I put my mind to it , I can accomplish things too . And I think that 's good , but , um , I think when you push , you know , maybe I think when that was first trying come about with , you know , what we know as the women 's lib movement -- -- I think it was too extreme . Um , I think you can be feminine and still be all those other things too , you know , and I think there 's definite roles , and I think , you know , part of the break up of the family , maybe it 's because of the , the , the social changes , I do n't know , the fact that , uh , during their early changes where , you know , women did come more , speak , maybe speak out , I was going to say , come more out of themselves , - but speak out , - I think a lot of me\n",
      "> 4114-73 0.125 Yeah , there , it 's almost all automobiles because there 's not that much in the way of -- -- heavy industry , you know , that would be causing it .\n",
      "> 2232-38 0.127 Uh - huh . Still have a problem with , uh , - you know , I have n't come to an absolute conclusion on my opinion on this , but , and I know other Christians would disagree with me . My husband and I , are kind of not even in agreement on this , but we do n't fight over it or anything , but , you know , how can , - you know , the Bible says bless your enemies and bless those that curse you , and it 's like , be gentle unto all men , apt to teach , patient , kind , so it 's like how can you , - I do n't know , for me , - I do n't know , you know , I ca n't say that I agree with Vietnam , because how can you be gentle unto all men , and , and then shoot them . So , Uh - huh .\n",
      "> 2587-56 0.128 Well , the other thing I can think of is , uh , what is still going on in Iraq --\n",
      "> 3185-51 0.128 Well another reason , - well , I guess I ca n't say that it , I do n't like the rain completely . We have a little garden that we have and -- -- and so it really helps our garden , and , you know , even , I do n't know if that 's because plants are just that way and they like rain , or , it seems they thrive when it rains , they just , really , -\n",
      "\n",
      "context-utterances\n",
      ">> 2793-24 0.212 That 's true . Well , # I do n't , I do n't think that , # -\n",
      ">> 3489-52 0.235 Oh , yes , I sure do .\n",
      ">> 3345-91 0.259 Yeah , well , I think that , um , - I do n't know .\n",
      ">> 3696-21 0.262 Uh - huh , I think you 're right .\n",
      ">> 2818-125 0.273 You do n't think , you do n't think that good prevails .\n",
      ">> 2488-34 0.275 That 's right , I think you 're right .\n",
      ">> 4796-70 0.286 Yeah . I think you 're right .\n",
      ">> 2968-64 0.287 Yeah , uh - huh . Well , that , do not , -\n",
      ">> 4037-0 0.291 Uh , Greg , uh , I , I 'm not familiar . I think , uh , you guys in Indiana , do n't you have the , the death penalty ?\n",
      ">> 2565-20 0.293 Yeah , I think that 's right .\n",
      "\n",
      "====\n",
      "\n",
      "CLUSTER 1 1\n",
      "---\n",
      "terms\n",
      "       cluster_dist\n",
      "index              \n",
      "a          0.207781\n",
      "i          0.277988\n",
      "but        0.285680\n",
      "and        0.292972\n",
      "it         0.313837\n",
      "uh         0.327858\n",
      "yeah       0.329786\n",
      "of         0.332502\n",
      "you        0.340146\n",
      "the        0.340576\n",
      "\n",
      "context terms\n",
      "        cluster_dist\n",
      "index               \n",
      "huh         0.277571\n",
      "yeah        0.349547\n",
      "i           0.378299\n",
      "oh          0.453633\n",
      "well        0.519320\n",
      "you         0.590355\n",
      "sounds      0.601005\n",
      "uh          0.607363\n",
      "it          0.613506\n",
      "okay        0.674758\n",
      "\n",
      "\n",
      "utterances\n",
      "> 3681-27 0.130 Oh , well now , yes I do n't think you can blame the weather on that . We had . -\n",
      "> 2283-6 0.130 Well , that 's pretty good . My , my sister is very over zealous , too . She 's got some really nice flower beds . She puts a lot of time into them . This is my first year in a , in a house where I 'm thinking about doing some flower beds and stuff . Oh , we , I 've always had crude ones at my father 's , but -- -- this is the first time that I really have to landscape a house . His house is more like a farm house . It 's not on a main road . My house is on a main road , and it has nothing I mean , nothing . But , um , what I do do , and I 've always done it is , - at my father 's house , we have , my sister and I have a couple of gardens . I think we figured out total , this is for vegetables -- -- about two thirds of an acre . So we each have a third of an acre we do . That 's a lot\n",
      "> 2995-5 0.132 Oh , now see I , I really do n't have much trouble with , I just say no thank you and hang up .\n",
      "> 4319-7 0.133 I 've got a Bachelor 's in electrical engineering so , - And I 'm not , like , a hugely advanced degree or any ( ( of that stuff ) ) . Uh , anyway , the , our benefits are pretty good . We 've got stock purchase program and a , - that , that 's pretty generous , although you have to hold the stock for two years -- -- before you get the company contribution which is , kind of a , a pain , but , uh , there 's a four O one K plan -- -- uh , for , uh , sheltering some taxable income .\n",
      "> 2344-103 0.134 Uh - huh . Yeah , well , yeah , and that . -\n",
      "> 2631-227 0.134 Uh - huh , yeah , and I 've discussed ,\n",
      "> 2175-92 0.137 Not at all . But , see , um , when I was younger I used to say that I would like to have five . Well , I 've even cut that down now , and , um , but to me five was n't even a lot , because I was just so used to twelve .\n",
      "> 3142-96 0.137 Yeah , and do the dishes . - And , uh , actually it 's kind of nice to , you know , - I always , - uh , a waitress or waiter that waits the table , it 's nice that they 're attentive but it 's not nice when they 're too attentive . So , uh , we 've had experience with that , too . So , -\n",
      "> 3796-51 0.138 So it 's , yeah , it 's not really a local , you know , -\n",
      "> 3419-39 0.138 Yes . I have eaten there . Now the only problem , - * note the setup for a negative commentI love the food . It 's a lot of cajun food and -- -- good seafood , but , uh , the service has always been so strange , every time I 've been in there . Mostly it 's been at lunch time -- -- and , uh , but I hear at night though they have , uh , you know , the outdoor , um , bar , and , um , uh , that 's it 's really , uh , you know , kind of lively . So , it 's , it 's really good . All their blackened grill stuff .\n",
      "\n",
      "context-utterances\n",
      ">> 3728-80 0.217 Well , yeah , I do n't , I do n't , uh , -\n",
      ">> 2041-112 0.230 Uh - huh . Yeah , well I ,\n",
      ">> 2950-125 0.251 Well , I , that , that is really healthier , frankly .\n",
      ">> 2956-80 0.259 # Well , it , uh , - well , I do n't know . #\n",
      ">> 2744-162 0.262 # Uh - huh , well that 's , that 's it you , you do n't have , uh , #\n",
      ">> 3971-33 0.266 Yeah , right . Well , I prefer ba- , uh , football --\n",
      ">> 2181-107 0.266 You 're right , uh - huh , yeah , it 's true .\n",
      ">> 3035-12 0.268 Uh - huh . Oh , yeah . I , -\n",
      ">> 2019-25 0.271 Well that 's understandable , yeah , it 's , uh , -\n",
      ">> 2107-35 0.280 Yes , uh - huh , that 's it\n",
      "\n",
      "====\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ec_fw.print_clusters(corpus=sw_corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CLUSTER 0 0\n",
      "---\n",
      "terms\n",
      "        cluster_dist\n",
      "index               \n",
      "a           0.206770\n",
      "i           0.236567\n",
      "well        0.256482\n",
      "and         0.259060\n",
      "so          0.260796\n",
      "it          0.298410\n",
      "to          0.300810\n",
      "really      0.311325\n",
      "you         0.322042\n",
      "uh          0.335718\n",
      "\n",
      "context terms\n",
      "        cluster_dist\n",
      "index               \n",
      "huh         0.391466\n",
      "yeah        0.437190\n",
      "i           0.450316\n",
      "you         0.489002\n",
      "oh          0.560207\n",
      "that        0.561080\n",
      "well        0.575842\n",
      "it          0.608198\n",
      "sounds      0.639354\n",
      "okay        0.653139\n",
      "\n",
      "\n",
      "utterances\n",
      "> 3405-0 0.109 Okay , Lowell , so I 'd like to know , um , what , what do you do in lawn and garden , what , uh , what 's , what 's of interest to you and how do you go about it ?\n",
      "> 2709-79 0.111 So , and I really do wish I knew about quotas and really wish I -\n",
      "> 3014-69 0.114 So , I just trusted the CONSUMER REPORTS and the auto ,\n",
      "> 4378-11 0.116 Well , I , I just live in a , I live in an apartment now . I , uh , two summers ago I went to Massachusetts -- -- and I went with a friend of mine and we undertook a building house . And this was a , a Lincoln log house where you have the wall partitions and it 's preconstructed you might say . And we started in , uh , we , started from an empty lot with , uh , trees and stuff on it and we had to cut them , uh , down and clear the lot . We had to call in the excavators and have them dig the basement . Pour the basement and , uh , went from the ground up . # Our # -- -- one of our main problems was , - well in Massachusetts , I thought I was going to , - well , it was a , a fun time # but # -- -- I thought , gosh , summer time , you know . Well , all it ever did was rain and thunder storms # # And one thing is that we 're , we had , we were going to pour the basement foundation . And we 're in the process and it started pouring down rain . And I guess we had most of it done but the end \n",
      "> 3725-18 0.116 Yeah , see that is kind of what happened with ours . That is why we have not had one in a long time . # So , # so , to make one successful , I mean , I mean what do you all do . Do you all just start planning real far ahead of time ? Uh , do you all start planning real far ahead of time ?\n",
      "> 2548-47 0.119 and resale value is really important to me\n",
      "> 2938-168 0.121 I got , I , I 've got one vice and smoke is , smoking is it . I do n't , I do n't drink , and I try not to cuss , and I do , I do very little , and smoking , I just , I got in the habit of it when I was about thirteen .\n",
      "> 2292-12 0.122 Yeah , I do n't ever want to have to worry about that . That 's real important to me . Um , you know we have that , that Aetna -- -- that 's what we , the insurance that we have right now .\n",
      "> 2344-103 0.123 Uh - huh . Yeah , well , yeah , and that . -\n",
      "> 3754-31 0.125 I loved it . I absolutely loved it . Yeah , and , uh , -\n",
      "\n",
      "context-utterances\n",
      ">> 2956-80 0.212 # Well , it , uh , - well , I do n't know . #\n",
      ">> 3728-80 0.229 Well , yeah , I do n't , I do n't , uh , -\n",
      ">> 2181-107 0.248 You 're right , uh - huh , yeah , it 's true .\n",
      ">> 3489-52 0.261 Oh , yes , I sure do .\n",
      ">> 3368-89 0.267 Oh , yeah , I suppose you do .\n",
      ">> 2982-31 0.273 Right , yeah , but I do n't , -\n",
      ">> 4908-4 0.273 But I gather you do not separate , uh , the bottles and cans .\n",
      ">> 3069-105 0.277 Yeah . Oh , that 's what I do too .\n",
      ">> 2968-64 0.284 Yeah , uh - huh . Well , that , do not , -\n",
      ">> 2275-148 0.286 Well , I think I 've , um , - you 're the second female I 've talked to .\n",
      "\n",
      "====\n",
      "\n",
      "CLUSTER 1 1\n",
      "---\n",
      "terms\n",
      "          cluster_dist\n",
      "index                 \n",
      "just          0.232631\n",
      "but           0.255032\n",
      "me            0.310694\n",
      "for           0.346174\n",
      "do            0.355607\n",
      "of            0.361877\n",
      "guess         0.368088\n",
      "any           0.375539\n",
      "not           0.378226\n",
      "probably      0.383363\n",
      "\n",
      "context terms\n",
      "       cluster_dist\n",
      "index              \n",
      "do         0.443428\n",
      "think      0.586346\n",
      "true       0.663341\n",
      "know       0.720145\n",
      "what       0.723235\n",
      "sure       0.752637\n",
      "how        0.837393\n",
      "not        0.906378\n",
      "is         0.936014\n",
      "mean       0.942061\n",
      "\n",
      "\n",
      "utterances\n",
      "> 2012-82 0.091 You know , it very much is . But on the other hand , I realized I could go out on the street and act like a complete lunatic , and people would leave , me alone . Whereas in this country where everyone respects the closed doors very much , if you go out and then act like the lunatic you , you violate the , uh , the norms of social , of , um , public behavior . Um , people start paying attention to you very much and they start asking questions and in the sense are invite- , invade , invading your privacy although , if you know what the social norms are , you know , quote unquote , you asked for it . But it does mean that you have yet another reason to follow a set of social norms . And which is n't of , always the case in all cultures , and it was n't until I was thinking about it just now that I realized that 's actually something that 's culturally relative .\n",
      "> 2379-94 0.093 Yeah . Well , I 'd , I 'd say actually I mean as someone who 's involved in education I 'd say that maybe one of the changes is that the , role of the teacher has incrementally gotten lower and lower value and society . I mean , relative pay which is a major way we value people , has been poorer and poorer over the years . Uh , you know , that - there 's been cost of living increases but not quite in proportion to cost of living and , you know it 's just more and more , uh , a real low income sort of job and very low prestige . I mean , there 's that , that old saying those who can do it and those who ca n't teach , this is the way of let 's make fun of the teachers those are the people who ca n't do anything . And if you have that kind of social attitude it 's hard to get sufficient numbers of people who are going to overlook all of that work for lower pay , work for low social prestige , just because they care about good education . And you get more and more people ( ( ) ) and up the\n",
      "> 2431-130 0.094 So , uh , and I think , I , I believe safety , - I , I , I , I really do believe in this stuff , uh , and I , I think it can go , ( ( ) ) , - I 'm not , - the air bags are a good deal but , uh , surprisingly , uh , they 're , uh , - you really need to do , - you need a combination of both the air bags and the , uh ,\n",
      "> 2959-180 0.097 Yeah . Well , you see that is what the veterans are so angry about now . They say , \" Well , we 're , going to listen to this and we are going to look at this , find out what our mistakes were , and we are not going to make them again . \" But that does not undo the ones that were done . And that is what makes the veterans angry . And I can not blame them . I do not know a solution . But I sure can not blame them for being angry . On the other hand , you know , you can be angry about something for a very long time , or you can say , \" Well , that is the way it is , \" and go on with your life . Uh , For the most part , I wish that it had not happened . But it did .\n",
      "> 2040-20 0.099 Right . What about if , if , um , they demanded to have Spanish as the official language as a condition for statehood .\n",
      "> 2759-126 0.101 Yeah So , I guess if they perfect that ( ( ) ) is not quite as cumbersome as having to go fill a bottle or --\n",
      "> 2060-10 0.102 Yeah , well , I know even if you watched A B C , N B C or the other , I mean , what 's the other one , uh , # C B S. # They all were , were tapped into C N N. That 's the only thing they broadcast and like you I listened to radio on my job at work . And all this week they have been having this , uh , a discussion about that is why , uh , why C N N was - well , I listen to a Christian radio station . And they were saying that C N N is definitely a world , uh , news service and , uh , they , - it was , - you ca n't really be sure of the quality of what you 've got , you know . Uh , we had some , uh , some people from our church went to Israel , uh , just for a , uh , tour sort of thing . And I was watching on TV they , they broadcast this terrible riot supposedly that was going on in Jerusalem , and so it really made me question as to what , how do we know , you know , uh , if the news we 're getting is any good . So -\n",
      "> 2314-64 0.106 Well , no , I do n't think they can , they can force another company to not drug test me just by saying that I , I -- -- did n't , I mean , - they do n't know that I do n't use drugs . They just tested me once . But , I used to work for a power company , so it was very important that they make sure their employees , especially linemen -- -- uh , were clean . And since I was just , you know , one of the office folk , I guess it was n't as important to them that they test me regularly . But I know they test most of the service people fairly regularly -- -- reg- , uh , just across the board .\n",
      "> 4092-121 0.107 Uh - huh . Well are you into that that rock music , you know , all that druggy stuff and all that I mean ? You know what I am saying like , I mean like they come on the stage and like , you know , you , - they do n't even know they are there . You know . They are just so # out of it . #\n",
      "> 3097-105 0.108 Right , and , I think that that 's a , that 's a symptom of our society , it 's a symptom that there 's , there 's a general illness that needs to be healed . I do not know what the healing process would be or what was causing the problem , I am not , I am not God . But , I , I do know that in raising my boys , I really did meet a lot of pressures saying , you know , you need to get them into music lessons , you need to get them ,\n",
      "\n",
      "context-utterances\n",
      ">> 3074-46 0.187 Sure you do , uh - huh .\n",
      ">> 3381-23 0.239 Do you think that 's a deterrent ?\n",
      ">> 3658-18 0.258 I do n't think you , uh , -\n",
      ">> 2488-34 0.270 That 's right , I think you 're right .\n",
      ">> 4796-70 0.273 Yeah . I think you 're right .\n",
      ">> 3696-21 0.274 Uh - huh , I think you 're right .\n",
      ">> 2818-125 0.275 You do n't think , you do n't think that good prevails .\n",
      ">> 2793-24 0.279 That 's true . Well , # I do n't , I do n't think that , # -\n",
      ">> 2584-186 0.282 No I do n't blame you , not nowadays .\n",
      ">> 2950-71 0.286 Uh , I do not know what , what it is percentage wise but I , -\n",
      "\n",
      "====\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ec_bk.print_clusters(corpus=sw_corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "demo continues below"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that in each case, two clusters emerge that roughly correspond to utterances recounting personal experiences, and those providing commentary, generally not about personal matters. We'll label them as such, noting that there's a roughly 50-50 split with slightly more \"personal\" utterances than \"commentary\" ones:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ec_fw.set_cluster_names(['commentary','personal'])\n",
    "ec_bk.set_cluster_names(['personal', 'commentary'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>utts</th>\n",
       "      <th>terms</th>\n",
       "      <th>context_utts</th>\n",
       "      <th>context_terms</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>commentary</th>\n",
       "      <td>0.423153</td>\n",
       "      <td>0.461942</td>\n",
       "      <td>0.404751</td>\n",
       "      <td>0.435696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>personal</th>\n",
       "      <td>0.576847</td>\n",
       "      <td>0.538058</td>\n",
       "      <td>0.595249</td>\n",
       "      <td>0.564304</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                utts     terms  context_utts  context_terms\n",
       "commentary  0.423153  0.461942      0.404751       0.435696\n",
       "personal    0.576847  0.538058      0.595249       0.564304"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ec_fw.print_cluster_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpreting derived characterizations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`ec_fw` and `ec_bk` also compute term-level statistics that we refer to as (forwards or backwards) _ranges_, which we roughly interpret as modeling the strengths of our forwards expectations of the replies that a term tends to get, or the backwards expectations of the predecessors that the term tends to follow. To examine these statistics, we'll put them in a Pandas dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "term_df = pd.DataFrame({'index': ec_fw.get_terms(),\n",
    "                       'fw_range': ec_fw.get_term_ranges(),\n",
    "                       'bk_range': ec_bk.get_term_ranges()}).set_index('index')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fw_range</th>\n",
       "      <th>bk_range</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>have</th>\n",
       "      <td>0.826351</td>\n",
       "      <td>0.809287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>so</th>\n",
       "      <td>0.824033</td>\n",
       "      <td>0.816046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>around</th>\n",
       "      <td>0.825741</td>\n",
       "      <td>0.829453</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>thinking</th>\n",
       "      <td>0.807137</td>\n",
       "      <td>0.800061</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>full</th>\n",
       "      <td>0.807683</td>\n",
       "      <td>0.798818</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          fw_range  bk_range\n",
       "index                       \n",
       "have      0.826351  0.809287\n",
       "so        0.824033  0.816046\n",
       "around    0.825741  0.829453\n",
       "thinking  0.807137  0.800061\n",
       "full      0.807683  0.798818"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "term_df.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the characterizations derived from `ec_fw` and `ec_bk` are comparable (as we initialized the latter model with the former), we can also compare characterizations across the two models. In the later analysis, we'll examine two:\n",
    "* orientation: this statistic compares the relative magnitude of forwards and backwards ranges. In a [counseling conversation setting](https://www.cs.cornell.edu/~cristian/Orientation_files/orientation-forwards-backwards.pdf) we interpreted orientation as a measure of the relative extent to which an interlocutor aims to advance the conversation forwards with a term, versus address existing content.\n",
    "* shift: this statistic corresponds to the distance between the backwards and forwards representations for each term; we interpret it as the extent to which a term shifts the focus of a conversation. \n",
    "\n",
    "These statistics are admittedly somewhat hard to interpret in the Switchboard setting, perhaps due to the relative lack of structures in these conversations. As we show later on, at the utterance level, they do bear some correspondence to various discourse act labels, so it might be worth playing around and coming up with characterizations of your own, that might reflect better-founded ideas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "term_df['orn'] = term_df.bk_range - term_df.fw_range\n",
    "term_df['shift'] = paired_distances(\n",
    "        ec_fw.ec_model.term_reprs, ec_bk.ec_model.term_reprs\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "low orientation\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>orn</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>how</th>\n",
       "      <td>-0.056177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>done</th>\n",
       "      <td>-0.055084</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>let</th>\n",
       "      <td>-0.050308</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>few</th>\n",
       "      <td>-0.049332</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>basically</th>\n",
       "      <td>-0.047425</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nothing</th>\n",
       "      <td>-0.045719</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nine</th>\n",
       "      <td>-0.043703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>okay</th>\n",
       "      <td>-0.043319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>whole</th>\n",
       "      <td>-0.042495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>works</th>\n",
       "      <td>-0.042461</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                orn\n",
       "index              \n",
       "how       -0.056177\n",
       "done      -0.055084\n",
       "let       -0.050308\n",
       "few       -0.049332\n",
       "basically -0.047425\n",
       "nothing   -0.045719\n",
       "nine      -0.043703\n",
       "okay      -0.043319\n",
       "whole     -0.042495\n",
       "works     -0.042461"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "high orientation\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>orn</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>taking</th>\n",
       "      <td>0.036944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>makes</th>\n",
       "      <td>0.037245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>change</th>\n",
       "      <td>0.038951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>funny</th>\n",
       "      <td>0.039011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>next</th>\n",
       "      <td>0.039310</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>called</th>\n",
       "      <td>0.043352</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>least</th>\n",
       "      <td>0.051271</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>anymore</th>\n",
       "      <td>0.051410</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>seem</th>\n",
       "      <td>0.064159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>myself</th>\n",
       "      <td>0.064653</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              orn\n",
       "index            \n",
       "taking   0.036944\n",
       "makes    0.037245\n",
       "change   0.038951\n",
       "funny    0.039011\n",
       "next     0.039310\n",
       "called   0.043352\n",
       "least    0.051271\n",
       "anymore  0.051410\n",
       "seem     0.064159\n",
       "myself   0.064653"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "low shift\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shift</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>he</th>\n",
       "      <td>0.162183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>to</th>\n",
       "      <td>0.173862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>was</th>\n",
       "      <td>0.176154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>car</th>\n",
       "      <td>0.195208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>read</th>\n",
       "      <td>0.208384</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>him</th>\n",
       "      <td>0.214645</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>we</th>\n",
       "      <td>0.216772</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>for</th>\n",
       "      <td>0.218534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>kids</th>\n",
       "      <td>0.223190</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>watch</th>\n",
       "      <td>0.223844</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          shift\n",
       "index          \n",
       "he     0.162183\n",
       "to     0.173862\n",
       "was    0.176154\n",
       "car    0.195208\n",
       "read   0.208384\n",
       "him    0.214645\n",
       "we     0.216772\n",
       "for    0.218534\n",
       "kids   0.223190\n",
       "watch  0.223844"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "high shift\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shift</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>goes</th>\n",
       "      <td>1.104640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>how</th>\n",
       "      <td>1.107404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>exactly</th>\n",
       "      <td>1.117140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>least</th>\n",
       "      <td>1.142747</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>change</th>\n",
       "      <td>1.162627</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>taking</th>\n",
       "      <td>1.206768</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>before</th>\n",
       "      <td>1.223504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nothing</th>\n",
       "      <td>1.270712</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>without</th>\n",
       "      <td>1.271164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tell</th>\n",
       "      <td>1.361794</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            shift\n",
       "index            \n",
       "goes     1.104640\n",
       "how      1.107404\n",
       "exactly  1.117140\n",
       "least    1.142747\n",
       "change   1.162627\n",
       "taking   1.206768\n",
       "before   1.223504\n",
       "nothing  1.270712\n",
       "without  1.271164\n",
       "tell     1.361794"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "k=10\n",
    "print('low orientation')\n",
    "display(term_df.sort_values('orn').head(k)[['orn']])\n",
    "print('high orientation')\n",
    "display(term_df.sort_values('orn').tail(k)[['orn']])\n",
    "print('\\nlow shift')\n",
    "display(term_df.sort_values('shift').head(k)[['shift']])\n",
    "print('high shift')\n",
    "display(term_df.sort_values('shift').tail(k)[['shift']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deriving utterance-level representations\n",
    "\n",
    "We now use the `ec_fw` and `ec_bk` models to derive utterance-level characterizations, by transforming the corpus with them. Again, we focus on utterances that are sufficiently long:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "_ = ec_fw.transform(sw_corpus, selector=lambda x: x.meta.get('col_normed_tfidf__n_feats',0)>=5)\n",
    "_ = ec_bk.transform(sw_corpus, selector=lambda x: x.meta.get('col_normed_tfidf__n_feats',0)>=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `transform` function does the following. \n",
    "\n",
    "First, it derives vector representations of utterances, stored as `fw_repr` and `bk_repr`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bk_repr', 'col_normed_tfidf', 'fw_repr'}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sw_corpus.vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, it derives ranges of utterances, stored in the metadata as `fw_range` and `bk_range`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Forwards range: 0.8370251853798225\n",
      "Backwards range: 0.8160712383960356\n"
     ]
    }
   ],
   "source": [
    "eg_ut = sw_corpus.get_utterance(utt_eg_id)\n",
    "print('Forwards range:', eg_ut.meta['fw_range'])\n",
    "print('Backwards range:', eg_ut.meta['bk_range'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It also assigns utterances to inferred types:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Forwards cluster: personal\n",
      "Backwards cluster: personal\n"
     ]
    }
   ],
   "source": [
    "print('Forwards cluster:', eg_ut.meta['fw_clustering.cluster'])\n",
    "print('Backwards cluster:', eg_ut.meta['bk_clustering.cluster'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with terms, we can derive orientations and shifts for utterances. For orientation, we compare the backwards and forwards ranges that the above calls to `transform` compute, for each utterance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for ut in sw_corpus.iter_utterances(selector=lambda x: x.meta.get('col_normed_tfidf__n_feats',0)>=5):\n",
    "    ut.meta['orn'] = ut.meta['bk_range'] - ut.meta['fw_range']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For shift, we compare the backwards and forwards representations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "utt_shifts = paired_distances(sw_corpus.get_vectors('fw_repr'), sw_corpus.get_vectors('bk_repr'))\n",
    "for id, shift in zip(sw_corpus.get_vector_matrix('fw_repr').ids, utt_shifts):\n",
    "    sw_corpus.get_utterance(id).meta['shift'] = shift"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shift: 0.6741247781441236\n",
      "orientation: -0.020953946983786942\n"
     ]
    }
   ],
   "source": [
    "print('shift:', eg_ut.meta['shift'])\n",
    "print('orientation:', eg_ut.meta['orn'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Analysis: correspondence to discourse act labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We explore the relation between the characterizations we've derived, and the various annotations that the utterances are labeled with (for more information on the annotation scheme, see the [manual here](https://web.stanford.edu/~jurafsky/ws97/manual.august1.html)). See [this dissertation](https://tisjune.github.io/research/dissertation) for further explanation of the analyses and findings below. A high-level comment is that this is a tough dataset for the framework to work with, given the relative lack of structure---something future work could think more carefully about.\n",
    "\n",
    "To facilitate the analysis, we extract relevant utterance attributes into a Pandas dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = sw_corpus.get_attribute_table('utterance',\n",
    "                ['bk_clustering.cluster', 'fw_clustering.cluster',\n",
    "                'orn', 'shift', 'tags'])\n",
    "df = df[df['bk_clustering.cluster'].notnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will stick to examining the 9 most common tags in the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tag_subset = ['aa', 'b', 'ba', 'h', 'ny', 'qw', 'qy', 'sd', 'sv'] \n",
    "for tag in tag_subset:\n",
    "    df['has_' + tag] = df.tags.apply(lambda x: tag in x.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start, we explore how the forwards and backwards vector representations correspond to these labels. To do this, we will compute log-odds ratios between the inferred utterance clusters and these labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def compute_log_odds(col, bool_col, val_subset=None):\n",
    "    if val_subset is not None:\n",
    "        col_vals = val_subset\n",
    "    else:\n",
    "        col_vals = col.unique()\n",
    "    log_odds_entries = []\n",
    "    for val in col_vals:\n",
    "        val_true = sum((col == val) & bool_col)\n",
    "        val_false = sum((col == val) & ~bool_col)\n",
    "        nval_true = sum((col != val) & bool_col)\n",
    "        nval_false = sum((col != val) & ~bool_col)\n",
    "        log_odds_entries.append({'val': val, 'log_odds': np.log((val_true/val_false)/(nval_true/nval_false))})\n",
    "    return log_odds_entries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bk_log_odds = []\n",
    "for tag in tag_subset:\n",
    "    entry = compute_log_odds(df['bk_clustering.cluster'],df['has_' + tag], ['commentary'])[0]\n",
    "    entry['tag'] = tag\n",
    "    bk_log_odds.append(entry)\n",
    "bk_log_odds_df = pd.DataFrame(bk_log_odds).set_index('tag').sort_values('log_odds')[['log_odds']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fw_log_odds = []\n",
    "for tag in tag_subset:\n",
    "    entry = compute_log_odds(df['fw_clustering.cluster'],df['has_' + tag], ['commentary'])[0]\n",
    "    entry['tag'] = tag\n",
    "    fw_log_odds.append(entry)\n",
    "fw_log_odds_df = pd.DataFrame(fw_log_odds).set_index('tag').sort_values('log_odds')[['log_odds']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "forwards types vs labels\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>tag</th>\n",
       "      <th>qy</th>\n",
       "      <th>ny</th>\n",
       "      <th>qw</th>\n",
       "      <th>sd</th>\n",
       "      <th>ba</th>\n",
       "      <th>b</th>\n",
       "      <th>aa</th>\n",
       "      <th>h</th>\n",
       "      <th>sv</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>log_odds</th>\n",
       "      <td>-0.491938</td>\n",
       "      <td>-0.485242</td>\n",
       "      <td>-0.472657</td>\n",
       "      <td>-0.380206</td>\n",
       "      <td>-0.304694</td>\n",
       "      <td>-0.159506</td>\n",
       "      <td>0.500705</td>\n",
       "      <td>0.718654</td>\n",
       "      <td>1.242259</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "tag             qy        ny        qw        sd        ba         b  \\\n",
       "log_odds -0.491938 -0.485242 -0.472657 -0.380206 -0.304694 -0.159506   \n",
       "\n",
       "tag             aa         h        sv  \n",
       "log_odds  0.500705  0.718654  1.242259  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------\n",
      "backwards types vs labels\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>tag</th>\n",
       "      <th>ny</th>\n",
       "      <th>qy</th>\n",
       "      <th>sd</th>\n",
       "      <th>ba</th>\n",
       "      <th>qw</th>\n",
       "      <th>b</th>\n",
       "      <th>aa</th>\n",
       "      <th>h</th>\n",
       "      <th>sv</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>log_odds</th>\n",
       "      <td>-0.56057</td>\n",
       "      <td>-0.428184</td>\n",
       "      <td>-0.416996</td>\n",
       "      <td>-0.344168</td>\n",
       "      <td>-0.32995</td>\n",
       "      <td>-0.142323</td>\n",
       "      <td>0.50554</td>\n",
       "      <td>0.733656</td>\n",
       "      <td>1.249692</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "tag            ny        qy        sd        ba       qw         b       aa  \\\n",
       "log_odds -0.56057 -0.428184 -0.416996 -0.344168 -0.32995 -0.142323  0.50554   \n",
       "\n",
       "tag              h        sv  \n",
       "log_odds  0.733656  1.249692  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print('forwards types vs labels')\n",
    "display(fw_log_odds_df.T)\n",
    "print('--------------------------')\n",
    "print('backwards types vs labels')\n",
    "display(bk_log_odds_df.T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tags further towards the right of the above tables (more positive log-odds) are those that co-occur more with the `commentary` than the `personal` utterance type. We briefly note that both forwards and backwards representations seem to draw a distinction between `sv` (opinion statements) and `sd` (non-opinion statements)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we explore how the orientation and shift statistics relate to these labels. To do this, we compare statistics for utterances with a particular label, to statistics for utterances without that label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def cohend(d1, d2):\n",
    "    n1, n2 = len(d1), len(d2)\n",
    "    s1, s2 = np.var(d1, ddof=1), np.var(d2, ddof=1)\n",
    "    s = np.sqrt(((n1 - 1) * s1 + (n2 - 1) * s2) / (n1 + n2 - 2))\n",
    "    u1, u2 = np.mean(d1), np.mean(d2)\n",
    "    return (u1 - u2) / s\n",
    "def get_pstars(p):\n",
    "    if p  < 0.001:\n",
    "        return '***'\n",
    "    elif p < 0.01:\n",
    "        return '**'\n",
    "    elif p < 0.05:\n",
    "        return '*'\n",
    "    else: return ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stat_col = 'orn'\n",
    "entries = []\n",
    "for tag in tag_subset:\n",
    "    has = df[df['has_' + tag]][stat_col]\n",
    "    hasnt = df[~df['has_' + tag]][stat_col]\n",
    "    entry = {'tag': tag, 'pval': stats.mannwhitneyu(has, hasnt)[1],\n",
    "            'cd': cohend(has, hasnt)}\n",
    "    entry['ps'] = get_pstars(entry['pval'] * len(tag_subset))\n",
    "    entries.append(entry)\n",
    "orn_stat_df = pd.DataFrame(entries).set_index('tag').sort_values('cd')\n",
    "orn_stat_df = orn_stat_df[np.abs(orn_stat_df.cd) >= .1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stat_col = 'shift'\n",
    "entries = []\n",
    "for tag in tag_subset:\n",
    "    has = df[df['has_' + tag]][stat_col]\n",
    "    hasnt = df[~df['has_' + tag]][stat_col]\n",
    "    entry = {'tag': tag, 'pval': stats.mannwhitneyu(has, hasnt)[1],\n",
    "            'cd': cohend(has, hasnt)}\n",
    "    entry['ps'] = get_pstars(entry['pval'] * len(tag_subset))\n",
    "    entries.append(entry)\n",
    "shift_stat_df = pd.DataFrame(entries).set_index('tag').sort_values('cd')\n",
    "shift_stat_df = shift_stat_df[np.abs(shift_stat_df.cd) >= .1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(We'll only show labels for which there's a sufficiently large difference, in cohen's delta, between utterances with and without the label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "orientation vs labels\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>tag</th>\n",
       "      <th>qw</th>\n",
       "      <th>sv</th>\n",
       "      <th>ba</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cd</th>\n",
       "      <td>-0.464487</td>\n",
       "      <td>0.137032</td>\n",
       "      <td>0.325546</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ps</th>\n",
       "      <td>***</td>\n",
       "      <td>***</td>\n",
       "      <td>***</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pval</th>\n",
       "      <td>4.57282e-58</td>\n",
       "      <td>3.04165e-41</td>\n",
       "      <td>1.04554e-35</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "tag            qw           sv           ba\n",
       "cd      -0.464487     0.137032     0.325546\n",
       "ps            ***          ***          ***\n",
       "pval  4.57282e-58  3.04165e-41  1.04554e-35"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------\n",
      "shift vs labels\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>tag</th>\n",
       "      <th>sd</th>\n",
       "      <th>h</th>\n",
       "      <th>sv</th>\n",
       "      <th>ny</th>\n",
       "      <th>qy</th>\n",
       "      <th>ba</th>\n",
       "      <th>qw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cd</th>\n",
       "      <td>-0.514413</td>\n",
       "      <td>-0.457883</td>\n",
       "      <td>-0.245104</td>\n",
       "      <td>-0.162611</td>\n",
       "      <td>0.199682</td>\n",
       "      <td>0.239983</td>\n",
       "      <td>0.493643</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ps</th>\n",
       "      <td>***</td>\n",
       "      <td>***</td>\n",
       "      <td>***</td>\n",
       "      <td>***</td>\n",
       "      <td>***</td>\n",
       "      <td>***</td>\n",
       "      <td>***</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pval</th>\n",
       "      <td>0</td>\n",
       "      <td>4.83333e-52</td>\n",
       "      <td>6.94752e-99</td>\n",
       "      <td>1.35491e-11</td>\n",
       "      <td>6.66404e-36</td>\n",
       "      <td>5.29252e-12</td>\n",
       "      <td>3.5594e-71</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "tag         sd            h           sv           ny           qy  \\\n",
       "cd   -0.514413    -0.457883    -0.245104    -0.162611     0.199682   \n",
       "ps         ***          ***          ***          ***          ***   \n",
       "pval         0  4.83333e-52  6.94752e-99  1.35491e-11  6.66404e-36   \n",
       "\n",
       "tag            ba          qw  \n",
       "cd       0.239983    0.493643  \n",
       "ps            ***         ***  \n",
       "pval  5.29252e-12  3.5594e-71  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print('orientation vs labels')\n",
    "display(orn_stat_df.T)\n",
    "print('--------------------------')\n",
    "print('shift vs labels')\n",
    "display(shift_stat_df.T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We note that utterances containing questions (`qw`, `qy`) have higher shifts than utterances which do not. If you're familiar with the DAMSL designations for forwards and backwards looking communicative functions, the output for orientation might look a little puzzling/informative that our view of what counts as forwards/backwards is different from the view espoused by the annotation scheme. We discuss this further in [this dissertation](https://tisjune.github.io/research/dissertation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Model persistence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we briefly demonstrate how the expected context models can be saved and loaded for later use. Here, we focus on `ec_fw`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "FW_MODEL_PATH = os.path.join(SW_CORPUS_PATH, 'fw')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ec_fw.dump(FW_MODEL_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In short, `ec_fw.dump` outputs latent context representations, clustering information, and various input parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "clustering_context_terms.tsv  cluster_names.npy  meta.json\r\n",
      "clustering_context_utts.tsv   context_s.npy      term_ranges.npy\r\n",
      "clustering_terms.tsv          context_terms.npy  term_reprs.npy\r\n",
      "clustering_utts.tsv           context_U.npy      terms.npy\r\n",
      "cluster_km_df.tsv             context_V.npy      train_utt_reprs.npy\r\n",
      "cluster_meta.json             km_model.joblib\r\n"
     ]
    }
   ],
   "source": [
    "ls $FW_MODEL_PATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To load the learned model, we start by initializing a new expected context model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ec_fw_new = ExpectedContextModelTransformer('next_id', 'fw_new', 'col_normed_tfidf', 'col_normed_tfidf', \n",
    "                                      n_svd_dims=15, n_clusters=2,\n",
    "                                     random_state=1000, cluster_random_state=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ec_fw_new.load(FW_MODEL_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that using the re-loaded model to transform the corpus results in the same representations as `ec_fw`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "_ = ec_fw_new.transform(sw_corpus, selector=lambda x: x.meta.get('col_normed_tfidf__n_feats',0)>=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(sw_corpus.get_vectors('fw_repr'), sw_corpus.get_vectors('fw_new_repr'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Pipeline usage\n",
    "\n",
    "We also implement a pipeline that handles the following:\n",
    "* processes text (via a pipeline supplied by the user)\n",
    "* transforms text to input representation (via `ColNormedTfidfTransformer`)\n",
    "* derives framework output (via `ExpectedContextModelTransformer`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from convokit.expected_context_framework import ExpectedContextModelPipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# see `demo_text_pipelines.py` in this demo's directory for details\n",
    "# in short, this pipeline will either output the `alpha_text` metadata  field\n",
    "# of an utterance, or write the utterance's `text` attribute into the `alpha_text` \n",
    "# metadata field\n",
    "from demo_text_pipelines import switchboard_text_pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We initialize the pipeline with the following arguments:\n",
    "* `text_field` specifies which utterance metadata field to use as text input\n",
    "* `text_pipe` specifies the pipeline used to compute the contents of `text_field`\n",
    "* `tfidf_params` specifies the parameters to be passed into the underlying `ColNormedTfidfTransformer` object\n",
    "* `min_terms` specifies the minimum number of terms in the vocabulary that an utterance must contain for it to be considered in fitting and transforming the underlying `ExpectedContextModelTransformer` object (see the `selector` argument passed into `ec_fw.fit` above)\n",
    "\n",
    "All other arguments are inherited from `ExpectedContextModelTransformer`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fw_pipe = ExpectedContextModelPipeline(context_field='next_id', output_prefix='fw',\n",
    "        text_field='alpha_text',\n",
    "        text_pipe=switchboard_text_pipeline(), \n",
    "        tfidf_params={'binary': True, 'vocabulary': vocab}, \n",
    "        min_terms=5,\n",
    "        n_svd_dims=15, n_clusters=2, cluster_on='utts',\n",
    "        random_state=1000, cluster_random_state=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# note this might output a warning that `col_normed_tfidf` already exists;\n",
    "# that's okay: the pipeline is just recomputing this matrix\n",
    "fw_pipe.fit(sw_corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with `ec_bk` we can initialize a second pipeline to compute backwards-characterizations, passing in argument `ec_model=fw_pipe` to ensure the derived representations in either direction are comparable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bk_pipe = ExpectedContextModelPipeline(context_field='reply_to', output_prefix='bk',\n",
    "        text_field='alpha_text',\n",
    "        text_pipe=switchboard_text_pipeline(), \n",
    "        tfidf_params={'binary': True, 'vocabulary': vocab}, \n",
    "        min_terms=5,\n",
    "        ec_model=fw_pipe,\n",
    "        n_svd_dims=15, n_clusters=2, cluster_on='utts',\n",
    "        random_state=1000, cluster_random_state=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bk_pipe.fit(sw_corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pipeline class inherits several other methods of `ExpectedContextModelTransformer`, e.g.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fw_pipe.set_cluster_names(['commentary','personal'])\n",
    "bk_pipe.set_cluster_names(['personal', 'commentary'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the pipeline enables us to transform ad-hoc string input: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "eg_ut_new = fw_pipe.transform_utterance('How old were you when you left ?')\n",
    "eg_ut_new = bk_pipe.transform_utterance(eg_ut_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, instead of storing vector representations with a corpus, the pipeline writes these representations to a field in the utterance metadata itself (since the utterance is not attached to a corpus):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.1269145021474405,\n",
       " -0.18177058434721982,\n",
       " 0.501745581507668,\n",
       " 0.3060047621669689,\n",
       " 0.18772232872850586,\n",
       " 0.30812601365393016,\n",
       " -0.17012829411583585,\n",
       " -0.5461342547631567,\n",
       " 0.24163519678100423,\n",
       " -0.07626502435693563,\n",
       " -0.23943785779562357,\n",
       " 0.10367101546307968,\n",
       " -0.11019044952480171,\n",
       " -0.06044023890156628]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eg_ut_new.meta['fw_repr']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Forwards range: 0.8370251853798226\n",
      "Backwards range: 0.8160712383960359\n",
      "Forwards cluster: personal\n",
      "Backwards cluster: personal\n"
     ]
    }
   ],
   "source": [
    "# note these attributes have the exact same values as those of eg_ut, computed above\n",
    "print('Forwards range:', eg_ut_new.meta['fw_range'])\n",
    "print('Backwards range:', eg_ut_new.meta['bk_range'])\n",
    "print('Forwards cluster:', eg_ut_new.meta['fw_clustering.cluster'])\n",
    "print('Backwards cluster:', eg_ut_new.meta['bk_clustering.cluster'])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}