vsm_02_dimreduce.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Vector-space models: dimensionality reduction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = \"Christopher Potts\"\n",
    "__version__ = \"CS224u, Stanford, Spring 2022\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Contents\n",
    "\n",
    "1. [Overview](#Overview)\n",
    "1. [Set-up](#Set-up)\n",
    "1. [Latent Semantic Analysis](#Latent-Semantic-Analysis)\n",
    "    1. [Overview of the LSA method](#Overview-of-the-LSA-method)\n",
    "    1. [Motivating example for LSA](#Motivating-example-for-LSA)\n",
    "    1. [Applying LSA to real VSMs](#Applying-LSA-to-real-VSMs)\n",
    "    1. [Other resources for matrix factorization](#Other-resources-for-matrix-factorization)\n",
    "1. [GloVe](#GloVe)\n",
    "    1. [Overview of the GloVe method](#Overview-of-the-GloVe-method)\n",
    "    1. [GloVe implementation notes](#GloVe-implementation-notes)\n",
    "    1. [Applying GloVe to our motivating example](#Applying-GloVe-to-our-motivating-example)\n",
    "    1. [Testing the GloVe implementation](#Testing-the-GloVe-implementation)\n",
    "    1. [Applying GloVe to real VSMs](#Applying-GloVe-to-real-VSMs)\n",
    "1. [Autoencoders](#Autoencoders)\n",
    "    1. [Overview of the autoencoder method](#Overview-of-the-autoencoder-method)\n",
    "    1. [Testing the autoencoder implementation](#Testing-the-autoencoder-implementation)\n",
    "    1. [Applying autoencoders to real VSMs](#Applying-autoencoders-to-real-VSMs)\n",
    "1. [Other methods](#Other-methods)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Overview\n",
    "\n",
    "The matrix weighting schemes reviewed in the first notebook for this unit deliver solid results. However, they are not capable of capturing higher-order associations in the data. \n",
    "\n",
    "With dimensionality reduction, the goal is to eliminate correlations in the input VSM and capture such higher-order notions of co-occurrence, thereby improving the overall space.\n",
    "\n",
    "As a motivating example, consider the adjectives _gnarly_ and _wicked_ used as slang positive adjectives.  Since both are positive, we expect them to be similar in a good VSM. However, at least stereotypically, _gnarly_ is Californian and _wicked_ is Bostonian. Thus, they are unlikely to occur often in the same texts, and so the methods we've reviewed so far will not be able to model their similarity. \n",
    "\n",
    "Dimensionality reduction techniques are often capable of capturing such semantic similarities (and have the added advantage of shrinking the size of our data structures)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Set-up\n",
    "\n",
    "* Make sure your environment meets all the requirements for [the cs224u repository](https://github.com/cgpotts/cs224u/). For help getting set-up, see [setup.ipynb](setup.ipynb).\n",
    "\n",
    "* Make sure you've downloaded [the data distribution for this course](http://web.stanford.edu/class/cs224u/data/data.tgz), unpacked it, and placed it in the current directory (or wherever you point `DATA_HOME` to below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch_glove import TorchGloVe\n",
    "import numpy as np\n",
    "from np_glove import GloVe\n",
    "import os\n",
    "import pandas as pd\n",
    "import scipy.stats\n",
    "from torch_autoencoder import TorchAutoencoder\n",
    "import utils\n",
    "import vsm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set all the random seeds for reproducibility:\n",
    "\n",
    "utils.fix_random_seeds()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_HOME = os.path.join('data', 'vsmdata')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "yelp5 = pd.read_csv(\n",
    "    os.path.join(DATA_HOME, 'yelp_window5-scaled.csv.gz'), index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "yelp20 = pd.read_csv(\n",
    "    os.path.join(DATA_HOME, 'yelp_window20-flat.csv.gz'), index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "giga5 = pd.read_csv(\n",
    "    os.path.join(DATA_HOME, 'giga_window5-scaled.csv.gz'), index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "giga20 = pd.read_csv(\n",
    "    os.path.join(DATA_HOME, 'giga_window20-flat.csv.gz'), index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Latent Semantic Analysis\n",
    "\n",
    "Latent Semantic Analysis (LSA) is a prominent dimensionality reduction technique. It is an application of __truncated singular value decomposition__ (SVD) and so uses only techniques from linear algebra (no machine learning needed)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Overview of the LSA method\n",
    "\n",
    "The central mathematical result is that, for any matrix of real numbers $X$ of dimension $m \\times n$, there is a factorization of $X$ into matrices $T$, $S$, and $D$ such that\n",
    "\n",
    "$$X_{m \\times n} = T_{m \\times m}S_{m\\times m}D_{n \\times m}^{\\top}$$\n",
    "\n",
    "The matrices $T$ and $D$ are  __orthonormal__ – their columns are length-normalized and orthogonal to one another (that is, they each have cosine distance of $1$ from each other). The singular-value matrix $S$ is a diagonal matrix arranged by size, so that the first dimension corresponds to the greatest source of variability in the data, followed by the second, and so on.\n",
    "\n",
    "Of course, we don't want to factorize and rebuild the original matrix, as that wouldn't get us anywhere. The __truncation__ part means that we include only the top $k$ dimensions of $S$. Given our row-oriented perspective on these matrices, this means using\n",
    "\n",
    "$$T[1{:}m, 1{:}k]S[1{:}k, 1{:}k]$$\n",
    "\n",
    "which gives us a version of $T$ that includes only the top $k$ dimensions of variation. \n",
    "\n",
    "To build up intuitions, imagine that everyone on the Stanford campus is associated with a 3d point representing their position: $x$ is east–west, $y$ is north–south, and $z$ is zenith–nadir. Since the campus is spread out and has relatively few deep basements and tall buildings, the top two dimensions of variation will be $x$ and $y$, and the 2d truncated SVD of this space will leave $z$ out. This will, for example, capture the sense in which someone at the top of Hoover Tower is close to someone at its base."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Motivating example for LSA\n",
    "\n",
    "We can also return to our original motivating example of _wicked_ and _gnarly_. Here is a matrix reflecting those assumptions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>gnarly</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wicked</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>awesome</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lame</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>terrible</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            0    1    2    3    4    5\n",
       "gnarly    1.0  0.0  1.0  0.0  0.0  0.0\n",
       "wicked    0.0  1.0  0.0  1.0  0.0  0.0\n",
       "awesome   1.0  1.0  1.0  1.0  0.0  0.0\n",
       "lame      0.0  0.0  0.0  0.0  1.0  1.0\n",
       "terrible  0.0  0.0  0.0  0.0  0.0  1.0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gnarly_df = pd.DataFrame(\n",
    "    np.array([\n",
    "        [1,0,1,0,0,0],\n",
    "        [0,1,0,1,0,0],\n",
    "        [1,1,1,1,0,0],\n",
    "        [0,0,0,0,1,1],\n",
    "        [0,0,0,0,0,1]], dtype='float64'),\n",
    "    index=['gnarly', 'wicked', 'awesome', 'lame', 'terrible'])\n",
    "\n",
    "gnarly_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No column context includes both _gnarly_ and _wicked_ together so our count matrix places them far apart:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gnarly      0.000000\n",
       "awesome     0.292893\n",
       "wicked      1.000000\n",
       "lame        1.000000\n",
       "terrible    1.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vsm.neighbors('gnarly', gnarly_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reweighting doesn't help. For example, here is the attempt with Positive PMI:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gnarly      0.000000\n",
       "awesome     0.292893\n",
       "wicked      1.000000\n",
       "lame        1.000000\n",
       "terrible    1.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vsm.neighbors('gnarly', vsm.pmi(gnarly_df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, both words tend to occur with _awesome_ and not with _lame_ or _terrible_, so there is an important sense in which they are similar. LSA to the rescue:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "gnarly_lsa_df = vsm.lsa(gnarly_df, k=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gnarly      0.0\n",
       "wicked      0.0\n",
       "awesome     0.0\n",
       "lame        1.0\n",
       "terrible    1.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vsm.neighbors('gnarly', gnarly_lsa_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Applying LSA to real VSMs\n",
    "\n",
    "Here's an example that begins to convey the effect that this can have empirically.\n",
    "\n",
    "First, the original count matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "superb         0.000000\n",
       "phenomenal     0.012159\n",
       "outstanding    0.012170\n",
       "fantastic      0.015080\n",
       "terrific       0.016488\n",
       "dtype: float64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vsm.neighbors('superb', yelp5).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then LSA with $k=100$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "yelp5_svd = vsm.lsa(yelp5, k=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "superb         0.000000\n",
       "outstanding    0.009402\n",
       "phenomenal     0.009575\n",
       "fantastic      0.012573\n",
       "terrific       0.014234\n",
       "dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vsm.neighbors('superb', yelp5_svd).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "A common pattern in the literature is to apply PMI first. The PMI values tend to give the count matrix a normal (Gaussian) distribution that better satisfies the assumptions underlying SVD:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "yelp5_pmi = vsm.pmi(yelp5, positive=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "yelp5_pmi_svd = vsm.lsa(yelp5_pmi, k=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "superb         0.000000\n",
       "phenomenal     0.040337\n",
       "outstanding    0.066857\n",
       "terrific       0.068770\n",
       "exceptional    0.077025\n",
       "dtype: float64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vsm.neighbors('superb', yelp5_pmi_svd).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Other resources for matrix factorization\n",
    "\n",
    "The [sklearn.decomposition](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.decomposition) module contains an implementation of LSA ([TruncatedSVD](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html#sklearn.decomposition.TruncatedSVD)) that you might want to switch to for real experiments:\n",
    "\n",
    "* The `sklearn` version is more flexible than the above in that it can operate on both dense matrices (Numpy arrays) and sparse matrices (from Scipy).\n",
    "\n",
    "* The `sklearn` version will make it easy to try out other dimensionality reduction methods in your own code; [Principal Component Analysis (PCA)](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html#sklearn.decomposition.PCA) and [Non-Negative Matrix Factorization (NMF)](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html#sklearn.decomposition.NMF) are closely related methods that are worth a look."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## GloVe\n",
    "\n",
    "### Overview of the GloVe method\n",
    "\n",
    "[Pennington et al. (2014)](http://www.aclweb.org/anthology/D/D14/D14-1162.pdf) introduce an objective function for semantic word representations. Roughly speaking, the objective is to learn vectors for words $w_{i}$ and $w_{j}$ such that their dot product is proportional to their probability of co-occurrence:\n",
    "\n",
    "$$w_{i}^{\\top}\\widetilde{w}_{k} + b_{i} + \\widetilde{b}_{k} = \\log(X_{ik})$$\n",
    "\n",
    "The paper is exceptionally good at motivating this objective from first principles. In their equation (6), they define \n",
    "\n",
    "$$w_{i}^{\\top}\\widetilde{w}_{k} = \\log(P_{ik}) = \\log(X_{ik}) - \\log(X_{i})$$\n",
    "\n",
    "If we allow that the rows and columns can be different, then we would do\n",
    "\n",
    "$$w_{i}^{\\top}\\widetilde{w}_{k} = \\log(P_{ik}) = \\log(X_{ik}) - \\log(X_{i} \\cdot X_{*k})$$\n",
    "\n",
    "where, as in the paper, $X_{i}$ is the sum of the values in row $i$, and $X_{*k}$ is the sum of the values in column $k$.\n",
    "\n",
    "The rightmost expression is PMI by the equivalence $\\log(\\frac{x}{y}) = \\log(x) - \\log(y)$, and hence we can see GloVe as aiming to make the dot product of two learned vectors equal to the PMI!\n",
    "\n",
    "The full model is a weighting of this objective:\n",
    "\n",
    "$$\\sum_{i, j=1}^{|V|} f\\left(X_{ij}\\right)\n",
    "  \\left(w_i^\\top \\widetilde{w}_j + b_i + \\widetilde{b}_j - \\log X_{ij}\\right)^2$$\n",
    "\n",
    "where $V$ is the vocabulary and $f$ is a scaling factor designed to diminish the impact of very large co-occurrence counts:\n",
    "\n",
    "$$f(x) \n",
    "\\begin{cases}\n",
    "(x/x_{\\max})^{\\alpha} & \\textrm{if } x < x_{\\max} \\\\\n",
    "1 & \\textrm{otherwise}\n",
    "\\end{cases}$$\n",
    "\n",
    "Typically, $\\alpha$ is set to $0.75$ and $x_{\\max}$ to $100$ (though it is worth assessing how many of your non-zero counts are above this; in dense word $\\times$ word matrices, you could be flattening more than you want to)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### GloVe implementation notes\n",
    "\n",
    "* The implementation in `np_glove.GloVe` is the most stripped-down, bare-bones version of the GloVe method I could think of. As such, it is quite slow. \n",
    "\n",
    "* `torch_glove.py` includes a vectorized implementation that is much, much faster, so we'll mainly use that. It is based on the method and implementation introduced in [Dingwall and Potts 2018](https://www.aclweb.org/anthology/N18-2034/).\n",
    "\n",
    "* For really large jobs, [the official C implementation released by the GloVe team](http://nlp.stanford.edu/projects/glove/) is probably the best choice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Applying GloVe to our motivating example\n",
    "\n",
    "GloVe should do well on our _gnarly/wicked_ evaluation, though you will see a lot variation due to the small size of this VSM:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "gnarly_glove_mod = GloVe(n=2, max_iter=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Finished epoch 999 of 1000; error is 3.371482852000653e-055"
     ]
    }
   ],
   "source": [
    "gnarly_glove = gnarly_glove_mod.fit(gnarly_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gnarly      0.000000\n",
       "terrible    0.115135\n",
       "wicked      0.151883\n",
       "lame        0.171145\n",
       "awesome     1.602795\n",
       "dtype: float64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vsm.neighbors('gnarly', gnarly_glove)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Testing the GloVe implementation\n",
    "\n",
    "It is not easy analyze GloVe values derived from real data, but the following little simulation suggests that `np_glove.glove` is working as advertised: it does seem to reliably deliver vectors whose dot products are proportional to the log co-occurrence probability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "glove_test_count_df = pd.DataFrame(\n",
    "    np.array([\n",
    "        [10.0,  2.0,  3.0,  4.0],\n",
    "        [ 2.0, 10.0,  4.0,  1.0],\n",
    "        [ 3.0,  4.0, 10.0,  2.0],\n",
    "        [ 4.0,  1.0,  2.0, 10.0]]),\n",
    "    index=['A', 'B', 'C', 'D'],\n",
    "    columns=['A', 'B', 'C', 'D'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "glove_test_mod = GloVe(n=4, max_iter=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Finished epoch 999 of 1000; error is 7.633385001520849e-052"
     ]
    }
   ],
   "source": [
    "glove_test_df = glove_test_mod.fit(glove_test_count_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `score` method for `GloVe` performs the correlation test that seems appropriate for GloVe. We can roughly summarize the test as follows: calculate the Pearson correlation of `G.dot(G.T)` and `log P(X)`, where `G` is the learned GloVe embedding and `log P(X)` gives the log probabilities. The test exempts `0` count cells when calculating the correlation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9566649559987249"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "glove_test_mod.score(glove_test_count_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Applying GloVe to real VSMs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `vsm.glove` implementation is too slow to use on real matrices. The version in the `torch_glove` module is significantly faster, making its use possible even without a GPU (and it will be very fast indeed on a GPU machine):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Finished epoch 100 of 100; error is 1671751.06875"
     ]
    }
   ],
   "source": [
    "glove_model = TorchGloVe(max_iter=100)\n",
    "\n",
    "yelp5_glv = glove_model.fit(yelp5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can again use the `score` method to see how we did:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.33100519572320014"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "glove_model.score(yelp5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "superb    0.000000\n",
       "stock     0.004879\n",
       "famous    0.004932\n",
       "posted    0.005239\n",
       "wonder    0.005317\n",
       "dtype: float64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vsm.neighbors('superb', yelp5_glv).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Autoencoders\n",
    "\n",
    "An autoencoder is a machine learning model that seeks to learn parameters that predict its own input. This is meaningful when there are intermediate representations that have lower dimensionality than the inputs. These provide a reduced-dimensional view of the data akin to those learned by LSA, but now we have a lot more design choices and a lot more potential to learn higher-order associations in the underyling data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Overview of the autoencoder method\n",
    "\n",
    "The module `torch_autoencoder` uses PyToch to implement a simple one-layer autoencoder:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "h &= \\mathbf{f}(xW + b_{h}) \\\\\n",
    "\\widehat{x} &= hW^{\\top} + b_{x}\n",
    "\\end{align}$$\n",
    "\n",
    "Here, we assume that the hidden representation $h$ has a low dimensionality like 100, and that $\\mathbf{f}$ is a non-linear activation function (the default for `TorchAutoencoder` is `tanh`). These are the major design choices internal to the network. It might also be meaningful to assume that there are two matrices of weights $W_{xh}$ and $W_{hx}$, rather than using $W^{\\top}$ for the output step.\n",
    "\n",
    "The objective function for autoencoders will implement some kind of assessment of the distance between the inputs and their predicted outputs. For example, one could use the one-half mean squared error:\n",
    "\n",
    "$$\\frac{1}{m}\\sum_{i=1}^{m} \\frac{1}{2}(\\widehat{X[i]} - X[i])^{2}$$\n",
    "\n",
    "where $X$ is the input matrix of examples (dimension $m \\times n$) and $X[i]$ corresponds to the $i$th example.\n",
    "\n",
    "When you call the `fit` method of `TorchAutoencoder`, it returns the matrix of hidden representations $h$, which is the new embedding space: same row count as the input, but with the column count set by the `hidden_dim` parameter.\n",
    "\n",
    "For much more on autoencoders, see the 'Autoencoders' chapter of [Goodfellow et al. 2016](http://www.deeplearningbook.org)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Testing the autoencoder implementation\n",
    "\n",
    "Here's an evaluation that is meant to test the autoencoder implementation – we expect it to be able to full encode the input matrix because we know its rank is equal to the dimensionality of the hidden representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def randmatrix(m, n, sigma=0.1, mu=0):\n",
    "    return sigma * np.random.randn(m, n) + mu\n",
    "\n",
    "def autoencoder_evaluation(nrow=1000, ncol=100, rank=20, max_iter=20000):\n",
    "    \"\"\"This an evaluation in which `TorchAutoencoder` should be able\n",
    "    to perfectly reconstruct the input data, because the\n",
    "    hidden representations have the same dimensionality as\n",
    "    the rank of the input matrix.\n",
    "    \"\"\"\n",
    "    X = randmatrix(nrow, rank).dot(randmatrix(rank, ncol))\n",
    "    ae = TorchAutoencoder(hidden_dim=rank, max_iter=max_iter)\n",
    "    ae.fit(X)\n",
    "    X_pred = ae.predict(X)\n",
    "    mse = (0.5 * (X_pred - X)**2).mean()\n",
    "    return(X, X_pred, mse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Finished epoch 100 of 100; error is 0.001447509159334004"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Autoencoder evaluation MSE after 100 evaluations: 0.0007\n"
     ]
    }
   ],
   "source": [
    "ae_max_iter = 100\n",
    "\n",
    "_, _, ae = autoencoder_evaluation(max_iter=ae_max_iter)\n",
    "\n",
    "print(\"Autoencoder evaluation MSE after {0} evaluations: {1:0.04f}\".format(\n",
    "    ae_max_iter, ae))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Applying autoencoders to real VSMs\n",
    "\n",
    "You can apply the autoencoder directly to the count matrix, but this could interact very badly with the internal activation function: if the counts are all very high or very low, then everything might get pushed irrevocably towards the extreme values of the activation.\n",
    "\n",
    "Thus, it's a good idea to first normalize the values somehow. Here, I use `vsm.length_norm`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "yelp5_l2 = yelp5.apply(vsm.length_norm, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Stopping after epoch 28. Training loss did not improve more than tol=1e-05. Final error is 0.00036822741822106764."
     ]
    }
   ],
   "source": [
    "yelp5_l2_ae = TorchAutoencoder(\n",
    "    max_iter=100, hidden_dim=50, eta=0.001).fit(yelp5_l2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "superb         0.000000\n",
       "outstanding    0.000018\n",
       "fantastic      0.000034\n",
       "terrific       0.000045\n",
       "exceptional    0.000050\n",
       "dtype: float64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vsm.neighbors('superb', yelp5_l2_ae).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is very slow and often seems not to work all that well. To speed things up, one can first apply LSA or similar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "yelp5_l2_svd100 = vsm.lsa(yelp5_l2, k=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Stopping after epoch 54. Training loss did not improve more than tol=1e-05. Final error is 0.001466967150918208."
     ]
    }
   ],
   "source": [
    "yelp_l2_svd100_ae = TorchAutoencoder(\n",
    "    max_iter=1000, hidden_dim=50, eta=0.01).fit(yelp5_l2_svd100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "superb         0.000000\n",
       "phenomenal     0.017522\n",
       "fantastic      0.020830\n",
       "terrific       0.025040\n",
       "outstanding    0.030703\n",
       "dtype: float64"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vsm.neighbors('superb', yelp_l2_svd100_ae).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Other methods\n",
    "\n",
    "Learning word representations is one of the most active areas in NLP right now, so I can't hope to offer a comprehensive summary. I'll settle instead for identifying some overall trends and methods:\n",
    "\n",
    "* The label __word2vec__ picks out a family of models in which the embedding for a word $w$ is trained to predict the words that co-occur with $w$. This intuition can be cashed out in numerous ways. The skip-gram model, due to [Mikolov et al. 2013](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality), is arguably the closest to the models discussed above. For instance, see [Levy and Golberg 2014](http://papers.nips.cc/paper/5477-neural-word-embedding-as-implicit-matrix-factorization) for a proof that the skip-gram model's objective reduces to PMI shifted by a constant value.\n",
    "\n",
    "* The LexVec model of [Salle et al. 2016](https://www.aclweb.org/anthology/P16-2068/) combines the core insight of GloVe (learn vectors that approximate PMI) with the insight from word2vec that we should additionally try to push words that don't appear together farther apart in the VSM. (GloVe simply ignores 0 count cells and so can't do this.)\n",
    "\n",
    "* Many apparently diverse models can be expressed as matrix factorization methods like SVD/LSA. See especially \n",
    "[Singh and Gordon 2008](http://www.cs.cmu.edu/~ggordon/singh-gordon-unified-factorization-ecml.pdf),\n",
    "[Levy and Golberg 2014](http://papers.nips.cc/paper/5477-neural-word-embedding-as-implicit-matrix-factorization), [Cotterell et al. 2017](https://www.aclweb.org/anthology/E17-2028/).\n",
    "\n",
    "* Subword modeling ([reviewed briefly in the previous notebook](vsm_01_distributional.ipynb#Subword-information)) is increasingly yielding dividends. (It would already be central if most of NLP focused on languages with complex morphology!) Check out the papers at the Subword and Character-Level Models for NLP Workshops: [SCLeM 2017](https://sites.google.com/view/sclem2017/home), [SCLeM 2018](https://sites.google.com/view/sclem2018/home).\n",
    "\n",
    "* Contextualized word representations have proven valuable in many contexts. These methods do not provide representations for individual words, but rather represent them in their linguistic context. This creates space for modeling how word senses vary depending on their context of use. See [vsm_03_contextualreps.ipynb](vsm_03_contextualreps.ipynb) for techniques for using such models to create VSMs like those explored above."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}