vsm_bakeoff_semantic_orientation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bake-off: The semantic orientation method\n",
    "\n",
    "__Important__: This isn't being run as a bake-off this year. It's included in the repository in case people want to do additional exploration or incorporate this kind of evaluation into a project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = \"Christopher Potts\"\n",
    "__version__ = \"CS224u, Stanford, Spring 2018 term\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "0. [Contents](#Contents)\n",
    "0. [Overview](#Overview)\n",
    "0. [Set-up](#Set-up)\n",
    "0. [Implementation](#Implementation)\n",
    "0. [Multidimensional sentiment lexicon](#Multidimensional-sentiment-lexicon)\n",
    "0. [Evaluation](#Evaluation)\n",
    "0. [Bake-off submission](#Bake-off-submission)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The semantic orientation method of [Turney and Littman 2003](http://doi.acm.org/10.1145/944012.944013) is a general, unsupervised (or lightly supervised?) method for building lexicons for any desired semantic dimension using VSMs.\n",
    "\n",
    "The method relies on intuitive seeds sets and the basic distance measures we use in basically all work with VSMs. Here's a summary of the method:\n",
    "\n",
    "1. Let $X$ be a VSM with dimension $m \\times n$ and vocabulary $V$. Let $I$ be the set of indices $\\{1, 2, \\ldots, m\\}$\n",
    "\n",
    "1. For $i \\in I$, let $X_{i}$ be the vector representation of $V_{i}$.\n",
    "\n",
    "1. Define two seed-sets $S{_1} \\subseteq I$ and $S_{2} \\subseteq I$. They should have the same cardinality and be semantically opposing in some way that is appropriate for your matrix.\n",
    "\n",
    "1. Pick a vector distance measure $\\textbf{f}$.\n",
    "\n",
    "1. For all $i \\in I$:\n",
    "\n",
    "$$\\textbf{score}(x_{i}) = \n",
    "\\left(\\sum_{j \\in S_{1}} \\textbf{f}(x_{i}, x_{j})\\right) \n",
    "- \n",
    "\\left(\\sum_{k \\in S_{2}} \\textbf{f}(x_{i}, s_{k})\\right)$$\n",
    "\n",
    "This method is implemented below as `semantic_orientation`. You can play around with making your own seed-sets and seeing what scores the method assigns. After that, the bake-off itself involves assessing your output against a multidimensional sentiment lexicon."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "import csv\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "from scipy.stats import pearsonr, spearmanr\n",
    "import vsm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_home = 'vsmdata'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def semantic_orientation(\n",
    "        df,        \n",
    "        seeds1=('bad', 'nasty', 'poor', 'negative', 'unfortunate', 'wrong', 'inferior'),\n",
    "        seeds2=('good', 'nice', 'excellent', 'positive', 'fortunate', 'correct', 'superior'),\n",
    "        distfunc=vsm.cosine):    \n",
    "    \"\"\"No frills implementation of the semantic Orientation (SO) method of \n",
    "    Turney and Littman. `seeds1` and `seeds2` should be representative members \n",
    "    of two intutively opposing semantic classes. The method will then try \n",
    "    to rank the vocabulary by its relative association with each seed set.\n",
    "        \n",
    "    Parameters\n",
    "    ----------\n",
    "    df : pd.DataFrame\n",
    "        The matrix used to derive the SO ranking.           \n",
    "    seeds1 : tuple of str\n",
    "        The default is the negative seed set of Turney and Littman.        \n",
    "    seeds2 : tuple of str\n",
    "        The default is the positive seed set of Turney and Littman.        \n",
    "    distfunc : function mapping vector pairs to floats (default: `cosine`)\n",
    "        The measure of distance between vectors. Can also be `euclidean`, \n",
    "        `matching`, `jaccard`, as well as any other distance measure \n",
    "        between 1d vectors. \n",
    "    \n",
    "    Returns\n",
    "    -------    \n",
    "    pd.Series\n",
    "        The vocabulary ranked according to the SO method, with words \n",
    "        closest to `seeds1` at the top and words closest to `seeds2` at the \n",
    "        bottom.\n",
    "    \n",
    "    \"\"\"\n",
    "    rownames = set(df.index)\n",
    "    # Check that the seed sets are in the vocabulary, filtering\n",
    "    # where necessary, and warn the user about exclusions:\n",
    "    seeds1 = _value_check(seeds1, \"seeds1\", rownames)\n",
    "    seeds2 = _value_check(seeds2, \"seeds2\", rownames)\n",
    "    \n",
    "    # Subframes for the two seeds-sets\n",
    "    sm1 = df.loc[seeds1]\n",
    "    sm2 = df.loc[seeds2]\n",
    "    \n",
    "    # Core semantic orientation calculation:\n",
    "    def row_func(row):\n",
    "        val1 = sm1.apply(lambda x: distfunc(row, x), axis=1).sum()\n",
    "        val2 = sm2.apply(lambda x: distfunc(row, x), axis=1).sum()\n",
    "        return val1 - val2\n",
    "    \n",
    "    scores = df.apply(row_func, axis=1)\n",
    "    return scores.sort_values(ascending=False)\n",
    "\n",
    "def _value_check(ss, name, rownames):\n",
    "    new = set()\n",
    "    for w in ss:\n",
    "        if w not in rownames:\n",
    "            print(\"Warning: {} not in {}\".format(w, name))\n",
    "        else:\n",
    "            new.add(w)\n",
    "    return new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "imdb20 = pd.read_csv(\n",
    "    os.path.join(data_home, 'imdb_window20-flat.csv.gz'), index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "imdb20_ppmi = vsm.pmi(imdb20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning: inferior not in seeds1\n"
     ]
    }
   ],
   "source": [
    "imdb20_ppmi_so = semantic_orientation(imdb20_ppmi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "excellent    0.596622\n",
       "superb       0.249968\n",
       "great        0.247541\n",
       "superior     0.230842\n",
       "nice         0.189436\n",
       "dtype: float64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imdb20_ppmi_so.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "unfortunate   -1.694198\n",
       "nasty         -1.838113\n",
       "poor          -1.907216\n",
       "wrong         -1.929000\n",
       "bad           -1.954924\n",
       "dtype: float64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imdb20_ppmi_so.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multidimensional sentiment lexicon\n",
    "\n",
    "[Warriner et al. (20130](http://www.humanities.mcmaster.ca/~vickup/Warriner-etal-BRM-2013.pdf) released a dataset called 'Norms of valence, arousal, and dominance for 13,915 English lemmas'. This is included in `vsmdata` as `Ratings_Warriner_et_al.csv`. The following code reads this file in and creates a DataFrame that gives just the overall means for these three semantic dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_warriner_lexicon(src_filename, df=None):\n",
    "    \"\"\"Read in 'Ratings_Warriner_et_al.csv' and optionally restrict its \n",
    "    vocabulary to items in `df`.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    src_filename : str\n",
    "        Full path to 'Ratings_Warriner_et_al.csv'\n",
    "    df : pd.DataFrame or None\n",
    "        If this is given, then its index is intersected with the \n",
    "        vocabulary from the lexicon, and we return a lexicon \n",
    "        containing only values in both vocabularies.\n",
    "        \n",
    "    Returns\n",
    "    -------\n",
    "    pd.DataFrame\n",
    "    \n",
    "    \"\"\"\n",
    "    lexicon = pd.read_csv(src_filename, index_col=0)\n",
    "    lexicon = lexicon[['Word', 'V.Mean.Sum', 'A.Mean.Sum', 'D.Mean.Sum']]\n",
    "    lexicon = lexicon.set_index('Word').rename(\n",
    "        columns={'V.Mean.Sum': 'Valence', \n",
    "                 'A.Mean.Sum': 'Arousal', \n",
    "                 'D.Mean.Sum': 'Dominance'})\n",
    "    if df is not None:\n",
    "        shared_vocab = sorted(set(lexicon.index) & set(df.index))\n",
    "        lexicon = lexicon.loc[shared_vocab]\n",
    "    return lexicon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "lexicon = load_warriner_lexicon(\n",
    "    os.path.join(data_home, 'Ratings_Warriner_et_al.csv'),\n",
    "    imdb20_ppmi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Valence</th>\n",
       "      <th>Arousal</th>\n",
       "      <th>Dominance</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Word</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>TV</th>\n",
       "      <td>5.42</td>\n",
       "      <td>4.29</td>\n",
       "      <td>6.23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ability</th>\n",
       "      <td>7.00</td>\n",
       "      <td>4.85</td>\n",
       "      <td>6.55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>able</th>\n",
       "      <td>6.64</td>\n",
       "      <td>3.38</td>\n",
       "      <td>6.17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>abortion</th>\n",
       "      <td>2.58</td>\n",
       "      <td>5.43</td>\n",
       "      <td>4.73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>absolute</th>\n",
       "      <td>5.43</td>\n",
       "      <td>3.48</td>\n",
       "      <td>5.58</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Valence  Arousal  Dominance\n",
       "Word                                 \n",
       "TV           5.42     4.29       6.23\n",
       "ability      7.00     4.85       6.55\n",
       "able         6.64     3.38       6.17\n",
       "abortion     2.58     5.43       4.73\n",
       "absolute     5.43     3.48       5.58"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lexicon.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "We evaluate VSMs by the Pearson correlation coefficient between the scores delivered by `semantic_orientation` and the values in the Warriner et al. lexicon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluation(lexicon, so, colname='Valence', metric=pearsonr):\n",
    "    lexicon['so'] = so\n",
    "    rho, pvalue = metric(lexicon['so'], lexicon[colname])\n",
    "    print(\"{0:}'s r: {1:0.3f}\".format(metric.__name__, rho))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline\n",
    "\n",
    "Here's a simple baseline: PPMI on `imdb20` as loaded above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning: inferior not in seeds1\n"
     ]
    }
   ],
   "source": [
    "imdb20_ppmi_so = semantic_orientation(imdb20_ppmi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pearsonr's r: 0.361\n"
     ]
    }
   ],
   "source": [
    "evaluation(lexicon, imdb20_ppmi_so, colname='Valence')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pearsonr's r: 0.005\n"
     ]
    }
   ],
   "source": [
    "evaluation(lexicon, imdb20_ppmi_so, colname='Arousal')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pearsonr's r: 0.315\n"
     ]
    }
   ],
   "source": [
    "evaluation(lexicon, imdb20_ppmi_so, colname='Dominance')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Bake-off submission\n",
    "\n",
    "1. The name of the count matrix you started with (must be one in `vsmdata`).\n",
    "1. The seed-sets you used.\n",
    "1. A description of the steps you took to create your bake-off VSM – must be different from the above baseline.\n",
    "1. Your Pearson r values for 'Valence', 'Arousal', and 'Dominance'."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}