07-produce-term-similarity-matrices.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d9d02ca5-eb70-4f90-b476-ee6298b5c2c7",
   "metadata": {},
   "source": [
    "# Produce term similarity matrices\n",
    "\n",
    "In this notebook, we will produce term similarity matrices to be used in the [soft vector space models][1].\n",
    "\n",
    " [1]: http://dx.doi.org/10.1145/3269206.3269317"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3f91ecd9-d2a7-438a-80f9-365dc1f9df09",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "docker.apollo.fi.muni.cz\n"
     ]
    }
   ],
   "source": [
    "! hostname"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d0036d6d-699e-4df5-ade9-3acd5cd2504e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! pip install .[scm]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "437cda96-b92c-4b1f-80d3-7f3a5e7e8437",
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.similarities import SparseTermSimilarityMatrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86e87b23-318d-4438-b836-55dc9de4f81a",
   "metadata": {},
   "source": [
    "## Levenshtein similarities\n",
    "\n",
    "First, we will produce term similarity matrices based on surface-level similarities using Levenshtein distances."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e57318c4-0654-4b66-8a5c-66fc779439a1",
   "metadata": {},
   "source": [
    "### The text + LaTeX format\n",
    "\n",
    "As our primary representation, we use text and LaTeX separated by special `[MATH]` and `[/MATH]` tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "64f9c1f0-b4c5-438a-ada7-48745b0144b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make levenshtein-similarity-matrix-text+latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "14dc5b0d-e922-4346-882d-d53e1670b7ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 1008 1011 38M May  6 18:50 levenshtein-similarity-matrix-text+latex\n"
     ]
    }
   ],
   "source": [
    "%ls -lh levenshtein-similarity-matrix-text+latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c29091b7-4b70-4838-81b6-8cbf13e72c85",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<71897x71897 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 4878801 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SparseTermSimilarityMatrix.load('levenshtein-similarity-matrix-text+latex').matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67fc93b0-11cf-40f5-b3fa-8f97302a7e0a",
   "metadata": {},
   "source": [
    "### The text format\n",
    "\n",
    "For baselines and for models with separate indices for text and math, we have a separate term similarity matrix with just text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "57aa66e5-1d40-41db-a0a5-8341b9654f82",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make levenshtein-similarity-matrix-text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b1a35301-b57f-4f3b-a189-6b27c7b4b957",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-r-- 1 1008 1011 26M May  6 11:53 levenshtein-similarity-matrix-text\n"
     ]
    }
   ],
   "source": [
    "%ls -lh levenshtein-similarity-matrix-text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ad02d471-aa36-45c7-b18b-113ed09fcf52",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<49559x49559 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 3276353 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SparseTermSimilarityMatrix.load('levenshtein-similarity-matrix-text').matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6dca405a-f922-42ad-adfa-a2df4f2a1efa",
   "metadata": {},
   "source": [
    "### The LaTeX format\n",
    "\n",
    "For baselines and for models with separate indices for text and math, we have a separate term similarity matrix with just LaTeX."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "644f7fb7-e165-4a29-bba4-a6181f1fa668",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make levenshtein-similarity-matrix-latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4b40272c-7767-45d1-8758-8e68354e673b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-r-- 1 1008 1011 16M May  6 09:14 levenshtein-similarity-matrix-latex\n"
     ]
    }
   ],
   "source": [
    "%ls -lh levenshtein-similarity-matrix-latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3711e28d-4910-4820-bdc9-45143deb260a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<29772x29772 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 2010880 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SparseTermSimilarityMatrix.load('levenshtein-similarity-matrix-latex').matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a4e3711-7894-4450-afcc-1fec3492fbc2",
   "metadata": {},
   "source": [
    "### The Tangent-L format\n",
    "\n",
    "For baselines and for models with separate indices for text and math, we have a separate term similarity matrix with just the format used by [the Tangent-L search engine from UWaterloo][1].\n",
    "\n",
    " [1]: http://ceur-ws.org/Vol-2936/paper-05.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "fab85f8e-ca22-4dd2-8bee-aa860301a5e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make levenshtein-similarity-matrix-tangentl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "192be600-347c-47ce-a012-739a86690e0d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 1008 1011 62M May  6 22:06 levenshtein-similarity-matrix-tangentl\n"
     ]
    }
   ],
   "source": [
    "%ls -lh levenshtein-similarity-matrix-tangentl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4544e4e2-4559-4e1c-8e4b-a39eb90037a5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<100000x100000 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 8065212 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SparseTermSimilarityMatrix.load('levenshtein-similarity-matrix-tangentl').matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a05a3f6c-3d76-4a17-bb68-57f05ec812cb",
   "metadata": {},
   "source": [
    "## Word embedding similarities\n",
    "\n",
    "Second, we will produce term similarity matrices based on semantic similarities using word embeddings extracted from language models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d31c695a-8a04-4422-888a-e46d910a0649",
   "metadata": {},
   "source": [
    "### Non-positional `word2vec` embeddings\n",
    "\n",
    "First, we will produce term similarity matrices based on semantic similarities using `word2vec` models without [positional weighting][1].\n",
    "\n",
    " [1]: https://github.com/MIR-MU/pine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c3d4788-3e79-4892-9bb3-1f48bc88dc0b",
   "metadata": {},
   "source": [
    "#### The text + LaTeX format\n",
    "\n",
    "As our primary representation, we use text and LaTeX separated by special `[MATH]` and `[/MATH]` tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "5510335c-e147-48af-961c-89cb9300372f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make word-embedding-similarity-matrix-text+latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "35aa21ec-8eaf-4607-8b52-36020ac62610",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 1008 1011 16M May  6 19:47 word-embedding-similarity-matrix-text+latex\n"
     ]
    }
   ],
   "source": [
    "%ls -lh word-embedding-similarity-matrix-text+latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "eac14ff2-9fad-49ea-8421-3d32f4ba6f10",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<71897x71897 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 2022487 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SparseTermSimilarityMatrix.load('word-embedding-similarity-matrix-text+latex').matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8775e35-a99e-4e78-9666-61cfd9968eee",
   "metadata": {},
   "source": [
    "#### The text format\n",
    "\n",
    "For models with separate indices for text and math, we have a separate term similarity matrix with just text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "12b1a5fb-e9c4-4db6-837b-a781d242d4a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make word-embedding-similarity-matrix-text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "6eecb2cd-2f16-49ed-81c2-ef8d20573dc9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 1008 1011 19M May  7 19:15 word-embedding-similarity-matrix-text\n"
     ]
    }
   ],
   "source": [
    "%ls -lh word-embedding-similarity-matrix-text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "dcc1a033-b3d5-40cf-97b4-e553a009e2db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<49559x49559 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 2444223 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SparseTermSimilarityMatrix.load('word-embedding-similarity-matrix-text').matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a666b937-9b81-46cd-9fdf-b604a3e72314",
   "metadata": {},
   "source": [
    "#### The LaTeX format\n",
    "\n",
    "For models with separate indices for text and math, we have a separate term similarity matrix with just LaTeX."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "7a487ff8-1ff0-4cba-af16-adba85a6161b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make word-embedding-similarity-matrix-latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "1afe8461-8a66-4a46-bb2b-d7eaf021038f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-r-- 1 1008 1011 6.0M May  6 12:03 word-embedding-similarity-matrix-latex\n"
     ]
    }
   ],
   "source": [
    "%ls -lh word-embedding-similarity-matrix-latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8388f015-d48a-436c-b495-3cfea1cc9ffc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<29772x29772 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 767746 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SparseTermSimilarityMatrix.load('word-embedding-similarity-matrix-latex').matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "229c1ef5-5800-43a1-9cf5-08b10eaececd",
   "metadata": {},
   "source": [
    "#### The Tangent-L format\n",
    "\n",
    "For models with separate indices for text and math, we have a separate term similarity matrix with just the format used by [the Tangent-L search engine from UWaterloo][1].\n",
    "\n",
    " [1]: http://ceur-ws.org/Vol-2936/paper-05.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "5f5f24cb-9193-4510-bf0a-0bb53d83fd82",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make word-embedding-similarity-matrix-tangentl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "1c58b66c-2161-4b8e-9f37-e8c94f0a947d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 1008 1011 5.8M May  6 23:45 word-embedding-similarity-matrix-tangentl\n"
     ]
    }
   ],
   "source": [
    "%ls -lh word-embedding-similarity-matrix-tangentl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "702d94cd-28e4-4c12-a861-5ec7de0afafe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<100000x100000 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 701998 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SparseTermSimilarityMatrix.load('word-embedding-similarity-matrix-tangentl').matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9072ed27-770e-4847-ab14-3beebf7f1b85",
   "metadata": {},
   "source": [
    "### Positional `word2vec` embeddings\n",
    "\n",
    "Next, we will produce term similarity matrices based on semantic similarities using `word2vec` models with [positional weighting][1].\n",
    "\n",
    " [1]: https://github.com/MIR-MU/pine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e2dcd54-1c1e-48b1-b38b-dfe56a4d3c0d",
   "metadata": {},
   "source": [
    "#### The text + LaTeX format\n",
    "\n",
    "As our primary representation, we use text and LaTeX separated by special `[MATH]` and `[/MATH]` tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "b23bf964-dd1c-4dae-bddd-0b2bc9621dc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make word-embedding-similarity-matrix-text+latex-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "2c63bd2a-61a2-4aea-8fd0-40d3af12313d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 1008 1011 22M May  6 19:48 word-embedding-similarity-matrix-text+latex-positional\n"
     ]
    }
   ],
   "source": [
    "%ls -lh word-embedding-similarity-matrix-text+latex-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "ad0e3494-6ff0-418e-9933-40ad1bcc9b65",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<71897x71897 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 2812669 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SparseTermSimilarityMatrix.load('word-embedding-similarity-matrix-text+latex-positional').matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d73f8c26-cfdd-436d-bb03-d21e6d7ca83c",
   "metadata": {},
   "source": [
    "#### The text format\n",
    "\n",
    "For baselines and for models with separate indices for text and math, we have a separate term similarity matrix with just text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "6b48e1a2-f9a6-4131-972f-b3adcad294b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make word-embedding-similarity-matrix-text-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "33317da8-d14b-4776-b432-c19f5ee27384",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 1008 1011 21M May  7 22:12 word-embedding-similarity-matrix-text-positional\n"
     ]
    }
   ],
   "source": [
    "%ls -lh word-embedding-similarity-matrix-text-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "211a0bfa-b529-4f84-9143-3e6a13d1eedb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<49559x49559 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 2657157 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SparseTermSimilarityMatrix.load('word-embedding-similarity-matrix-text-positional').matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c05a689-9bcd-4f37-a4b0-fc7e1ef63925",
   "metadata": {},
   "source": [
    "#### The LaTeX format\n",
    "\n",
    "For models with separate indices for text and math, we have a separate term similarity matrix with just LaTeX."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "4b964ddd-df5c-4f99-9c54-3a60a810d200",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make word-embedding-similarity-matrix-latex-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "4c64305a-1290-486b-8e51-caf739e4d6b1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-r-- 1 1008 1011 7.1M May  6 12:03 word-embedding-similarity-matrix-latex-positional\n"
     ]
    }
   ],
   "source": [
    "%ls -lh word-embedding-similarity-matrix-latex-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "bfdbcd30-5809-4558-9d5d-ab782d8fce9a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<29772x29772 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 906458 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SparseTermSimilarityMatrix.load('word-embedding-similarity-matrix-latex-positional').matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "047f3a14-b007-4132-bdb5-2a0096acd2c4",
   "metadata": {},
   "source": [
    "#### The Tangent-L format\n",
    "\n",
    "For models with separate indices for text and math, we have a separate term similarity matrix with just the format used by [the Tangent-L search engine from UWaterloo][1].\n",
    "\n",
    " [1]: http://ceur-ws.org/Vol-2936/paper-05.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "8e16d981-a393-40fc-9697-ad1b9ea24485",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make word-embedding-similarity-matrix-tangentl-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "d87ae242-703a-4494-93e6-0e484d34cef8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 1008 1011 9.7M May  6 23:38 word-embedding-similarity-matrix-tangentl-positional\n"
     ]
    }
   ],
   "source": [
    "%ls -lh word-embedding-similarity-matrix-tangentl-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "5375bfb5-1ade-4394-83e0-a061c0d4ef70",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<100000x100000 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 1216694 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SparseTermSimilarityMatrix.load('word-embedding-similarity-matrix-tangentl-positional').matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06ca9278-abb3-4a5e-ae34-b4b97d8edb82",
   "metadata": {},
   "source": [
    "### Decontextualized `roberta-base` embeddings\n",
    "\n",
    "Next, we will produce term similarity matrices based on semantic similarities using the [decontextualized word embeddings][1] of [the `roberta-base` model][2].\n",
    "\n",
    " [1]: https://aclanthology.org/2021.wmt-1.112\n",
    " [2]: https://huggingface.co/roberta-base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "a9efa9a2-f36c-418b-80e8-487a52031c61",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make decontextualized-word-embedding-similarity-matrix-roberta-base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "15e04dc7-6eb1-4cb4-8161-7aaec7ee4d4e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 1008 1011 720K May 25 17:45 decontextualized-word-embedding-similarity-matrix-roberta-base\n"
     ]
    }
   ],
   "source": [
    "%ls -lh decontextualized-word-embedding-similarity-matrix-roberta-base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "db27e623-dbc4-40d8-b904-6075352ebd4a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<49559x49559 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 67179 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SparseTermSimilarityMatrix.load('decontextualized-word-embedding-similarity-matrix-roberta-base').matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5049a9d-54c5-4bc5-b631-d8b2682a03a6",
   "metadata": {},
   "source": [
    "### Decontextualized tuned `roberta-base` embeddings\n",
    "\n",
    "Next, we will produce term similarity matrices based on semantic similarities using the [decontextualized word embeddings][1] of [the `roberta-base` model][2] fine-tuned so that it can represent math-specific tokens.\n",
    "\n",
    " [1]: https://aclanthology.org/2021.wmt-1.112\n",
    " [2]: https://huggingface.co/roberta-base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "6967b0cd-1292-406c-a210-2610facca29c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make decontextualized-word-embedding-similarity-matrix-tuned-roberta-base-text+latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "ac2381cf-3e8f-465a-b2b5-29553f7a0e0d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 1008 1011 1.3M May 25 17:49 decontextualized-word-embedding-similarity-matrix-tuned-roberta-base-text+latex\n"
     ]
    }
   ],
   "source": [
    "%ls -lh decontextualized-word-embedding-similarity-matrix-tuned-roberta-base-text+latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "62417762-e4c5-4617-b79d-270292087615",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<71897x71897 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 126649 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SparseTermSimilarityMatrix.load('decontextualized-word-embedding-similarity-matrix-tuned-roberta-base-text+latex').matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b823e394-eab6-4a25-9938-7f28014e39e2",
   "metadata": {},
   "source": [
    "## Combined similarities\n",
    "\n",
    "Through, we will combine the Levenshtein and word embedding term similarity matrices to create combined term similarity matrices that take into account both surface-level and semantic term similarities. We will produce combined term similarity matrices for different values of $\\alpha$ that will determine the relative weights of the two matrices. This will allow us to later optimize $\\alpha$ on our end task."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b41e5b80-dd16-4cec-b714-b3367993fd86",
   "metadata": {},
   "source": [
    "### Non-positional `word2vec` embeddings\n",
    "\n",
    "First, we will produce term similarity matrices based on semantic similarities using `word2vec` models without [positional weighting][1].\n",
    "\n",
    " [1]: https://github.com/MIR-MU/pine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e77be1ad-5117-4d0e-93ad-30667a819bac",
   "metadata": {},
   "source": [
    "#### The text + LaTeX format\n",
    "\n",
    "As our primary representation, we use text and LaTeX separated by special `[MATH]` and `[/MATH]` tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "7a61b07e-c0d5-4dc7-a365-482675fcc4be",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make similarity-matrix-text+latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "50a57750-fc3d-4e06-8db7-6366ea31d782",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 518M\n",
      "-rw-r--r-- 1 1008 1011 16M May 30 11:06 'alpha=0.0'\n",
      "-rw-r--r-- 1 1008 1011 52M May 30 11:06 'alpha=0.1'\n",
      "-rw-r--r-- 1 1008 1011 52M May 30 11:06 'alpha=0.2'\n",
      "-rw-r--r-- 1 1008 1011 52M May 30 11:06 'alpha=0.3'\n",
      "-rw-r--r-- 1 1008 1011 52M May 30 11:06 'alpha=0.4'\n",
      "-rw-r--r-- 1 1008 1011 52M May 30 11:06 'alpha=0.5'\n",
      "-rw-r--r-- 1 1008 1011 52M May 30 11:06 'alpha=0.6'\n",
      "-rw-r--r-- 1 1008 1011 52M May 30 11:06 'alpha=0.7'\n",
      "-rw-r--r-- 1 1008 1011 52M May 30 11:06 'alpha=0.8'\n",
      "-rw-r--r-- 1 1008 1011 52M May 30 11:06 'alpha=0.9'\n",
      "-rw-r--r-- 1 1008 1011 38M May 30 11:06 'alpha=1.0'\n"
     ]
    }
   ],
   "source": [
    "%ls -lh similarity-matrix-text+latex/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a96819b-26a9-4012-a3cc-9e2bdfd23b91",
   "metadata": {},
   "source": [
    "#### The text format\n",
    "\n",
    "For models with separate indices for text and math, we have a separate term similarity matrix with just text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "8d0b48db-35fa-4962-8745-fb447dce07ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make similarity-matrix-text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "54ab5e07-d7ba-4d2c-9161-94a7c95df65f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 431M\n",
      "-rw-r--r-- 1 1008 1011 19M May 30 11:06 'alpha=0.0'\n",
      "-rw-r--r-- 1 1008 1011 43M May 30 11:06 'alpha=0.1'\n",
      "-rw-r--r-- 1 1008 1011 43M May 30 11:06 'alpha=0.2'\n",
      "-rw-r--r-- 1 1008 1011 43M May 30 11:06 'alpha=0.3'\n",
      "-rw-r--r-- 1 1008 1011 43M May 30 11:06 'alpha=0.4'\n",
      "-rw-r--r-- 1 1008 1011 43M May 30 11:06 'alpha=0.5'\n",
      "-rw-r--r-- 1 1008 1011 43M May 30 11:06 'alpha=0.6'\n",
      "-rw-r--r-- 1 1008 1011 43M May 30 11:06 'alpha=0.7'\n",
      "-rw-r--r-- 1 1008 1011 43M May 30 11:06 'alpha=0.8'\n",
      "-rw-r--r-- 1 1008 1011 43M May 30 11:06 'alpha=0.9'\n",
      "-rw-r--r-- 1 1008 1011 26M May 30 11:06 'alpha=1.0'\n"
     ]
    }
   ],
   "source": [
    "%ls -lh similarity-matrix-text/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d50aeed-8170-4180-b996-ad3dbdae50b1",
   "metadata": {},
   "source": [
    "#### The LaTeX format\n",
    "\n",
    "For models with separate indices for text and math, we have a separate term similarity matrix with just LaTeX."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "adfee342-b9de-4883-a853-73632e346284",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make similarity-matrix-latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "c9970621-a75c-4c84-88e0-5af235dbff29",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 203M\n",
      "-rw-r--r-- 1 1008 1011 6.0M May 30 11:05 'alpha=0.0'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.1'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.2'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.3'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.4'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.5'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.6'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.7'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.8'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.9'\n",
      "-rw-r--r-- 1 1008 1011  16M May 30 11:05 'alpha=1.0'\n"
     ]
    }
   ],
   "source": [
    "%ls -lh similarity-matrix-latex/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37b7ff0d-fa6c-439f-b3be-174b14eef334",
   "metadata": {},
   "source": [
    "#### The Tangent-L format\n",
    "\n",
    "For models with separate indices for text and math, we have a separate term similarity matrix with just the format used by [the Tangent-L search engine from UWaterloo][1].\n",
    "\n",
    " [1]: http://ceur-ws.org/Vol-2936/paper-05.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "f5d63ab5-0a21-4e3b-9a51-f24394f5235f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make similarity-matrix-tangentl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "e2ff8692-3d07-40c8-a5ec-8b93e982ed85",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 648M\n",
      "-rw-r--r-- 1 1008 1011 5.8M May 30 11:05 'alpha=0.0'\n",
      "-rw-r--r-- 1 1008 1011  65M May 30 11:05 'alpha=0.1'\n",
      "-rw-r--r-- 1 1008 1011  65M May 30 11:05 'alpha=0.2'\n",
      "-rw-r--r-- 1 1008 1011  65M May 30 11:05 'alpha=0.3'\n",
      "-rw-r--r-- 1 1008 1011  65M May 30 11:05 'alpha=0.4'\n",
      "-rw-r--r-- 1 1008 1011  65M May 30 11:05 'alpha=0.5'\n",
      "-rw-r--r-- 1 1008 1011  65M May 30 11:06 'alpha=0.6'\n",
      "-rw-r--r-- 1 1008 1011  65M May 30 11:06 'alpha=0.7'\n",
      "-rw-r--r-- 1 1008 1011  65M May 30 11:06 'alpha=0.8'\n",
      "-rw-r--r-- 1 1008 1011  65M May 30 11:06 'alpha=0.9'\n",
      "-rw-r--r-- 1 1008 1011  62M May 30 11:06 'alpha=1.0'\n"
     ]
    }
   ],
   "source": [
    "%ls -lh similarity-matrix-tangentl/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdf7023b-e289-448d-b542-2d385e9facfa",
   "metadata": {},
   "source": [
    "### Positional `word2vec` embeddings\n",
    "\n",
    "Next, we will produce term similarity matrices based on semantic similarities using `word2vec` models with [positional weighting][1].\n",
    "\n",
    " [1]: https://github.com/MIR-MU/pine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb26c396-2000-4a7a-b5e5-2f07f60646cc",
   "metadata": {},
   "source": [
    "#### The text + LaTeX format\n",
    "\n",
    "As our primary representation, we use text and LaTeX separated by special `[MATH]` and `[/MATH]` tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "51dbafde-c8b1-489a-8341-0d2fc8f93ca2",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make similarity-matrix-text+latex-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "ccac432b-0625-460c-ab7c-23f36693e4e0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 577M\n",
      "-rw-r--r-- 1 1008 1011 22M May 30 11:06 'alpha=0.0'\n",
      "-rw-r--r-- 1 1008 1011 58M May 30 11:06 'alpha=0.1'\n",
      "-rw-r--r-- 1 1008 1011 58M May 30 11:06 'alpha=0.2'\n",
      "-rw-r--r-- 1 1008 1011 58M May 30 11:06 'alpha=0.3'\n",
      "-rw-r--r-- 1 1008 1011 58M May 30 11:06 'alpha=0.4'\n",
      "-rw-r--r-- 1 1008 1011 58M May 30 11:07 'alpha=0.5'\n",
      "-rw-r--r-- 1 1008 1011 58M May 30 11:07 'alpha=0.6'\n",
      "-rw-r--r-- 1 1008 1011 58M May 30 11:07 'alpha=0.7'\n",
      "-rw-r--r-- 1 1008 1011 58M May 30 11:07 'alpha=0.8'\n",
      "-rw-r--r-- 1 1008 1011 58M May 30 11:07 'alpha=0.9'\n",
      "-rw-r--r-- 1 1008 1011 38M May 30 11:07 'alpha=1.0'\n"
     ]
    }
   ],
   "source": [
    "%ls -lh similarity-matrix-text+latex-positional/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bb90e93-253e-4e72-9169-00657c3104c5",
   "metadata": {},
   "source": [
    "#### The text format\n",
    "\n",
    "For baselines and for models with separate indices for text and math, we have a separate term similarity matrix with just text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "457687ab-9fa7-498c-a145-e2c9a0486180",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make similarity-matrix-text-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "64f61f9b-b961-413f-a9fa-94197fa04867",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 446M\n",
      "-rw-r--r-- 1 1008 1011 21M May 30 11:07 'alpha=0.0'\n",
      "-rw-r--r-- 1 1008 1011 45M May 30 11:07 'alpha=0.1'\n",
      "-rw-r--r-- 1 1008 1011 45M May 30 11:07 'alpha=0.2'\n",
      "-rw-r--r-- 1 1008 1011 45M May 30 11:07 'alpha=0.3'\n",
      "-rw-r--r-- 1 1008 1011 45M May 30 11:07 'alpha=0.4'\n",
      "-rw-r--r-- 1 1008 1011 45M May 30 11:07 'alpha=0.5'\n",
      "-rw-r--r-- 1 1008 1011 45M May 30 11:07 'alpha=0.6'\n",
      "-rw-r--r-- 1 1008 1011 45M May 30 11:07 'alpha=0.7'\n",
      "-rw-r--r-- 1 1008 1011 45M May 30 11:07 'alpha=0.8'\n",
      "-rw-r--r-- 1 1008 1011 45M May 30 11:07 'alpha=0.9'\n",
      "-rw-r--r-- 1 1008 1011 26M May 30 11:07 'alpha=1.0'\n"
     ]
    }
   ],
   "source": [
    "%ls -lh similarity-matrix-text-positional/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd6f3f49-4717-4000-91cf-979b1ec73087",
   "metadata": {},
   "source": [
    "#### The LaTeX format\n",
    "\n",
    "For models with separate indices for text and math, we have a separate term similarity matrix with just LaTeX."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "d359b6c3-e769-40c1-ab65-80ae150a22c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make similarity-matrix-latex-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "7cc9f6be-fda4-4f9b-bcd2-8f039ff70a35",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 211M\n",
      "-rw-r--r-- 1 1008 1011 7.1M May 30 11:05 'alpha=0.0'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.1'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.2'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.3'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.4'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.5'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.6'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.7'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.8'\n",
      "-rw-r--r-- 1 1008 1011  21M May 30 11:05 'alpha=0.9'\n",
      "-rw-r--r-- 1 1008 1011  16M May 30 11:05 'alpha=1.0'\n"
     ]
    }
   ],
   "source": [
    "%ls -lh similarity-matrix-latex-positional/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2a4b6a6-9685-4529-b73b-319c49de01db",
   "metadata": {},
   "source": [
    "#### The Tangent-L format\n",
    "\n",
    "For models with separate indices for text and math, we have a separate term similarity matrix with just the format used by [the Tangent-L search engine from UWaterloo][1].\n",
    "\n",
    " [1]: http://ceur-ws.org/Vol-2936/paper-05.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "c89bee99-c977-4b36-bbfa-4522496b026d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make similarity-matrix-tangentl-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "f8b0f171-24d6-4183-8c98-8191ba3472ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 676M\n",
      "-rw-r--r-- 1 1008 1011 9.7M May 30 11:06 'alpha=0.0'\n",
      "-rw-r--r-- 1 1008 1011  68M May 30 11:06 'alpha=0.1'\n",
      "-rw-r--r-- 1 1008 1011  68M May 30 11:06 'alpha=0.2'\n",
      "-rw-r--r-- 1 1008 1011  68M May 30 11:06 'alpha=0.3'\n",
      "-rw-r--r-- 1 1008 1011  68M May 30 11:06 'alpha=0.4'\n",
      "-rw-r--r-- 1 1008 1011  68M May 30 11:06 'alpha=0.5'\n",
      "-rw-r--r-- 1 1008 1011  68M May 30 11:06 'alpha=0.6'\n",
      "-rw-r--r-- 1 1008 1011  68M May 30 11:06 'alpha=0.7'\n",
      "-rw-r--r-- 1 1008 1011  68M May 30 11:06 'alpha=0.8'\n",
      "-rw-r--r-- 1 1008 1011  68M May 30 11:06 'alpha=0.9'\n",
      "-rw-r--r-- 1 1008 1011  62M May 30 11:06 'alpha=1.0'\n"
     ]
    }
   ],
   "source": [
    "%ls -lh similarity-matrix-tangentl-positional/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbd1ce7c-d0d8-45b9-a138-09add0d5729e",
   "metadata": {},
   "source": [
    "### Decontextualized `roberta-base` embeddings\n",
    "\n",
    "Next, we will produce term similarity matrices based on semantic similarities using the [decontextualized word embeddings][1] of [the `roberta-base` model][2].\n",
    "\n",
    " [1]: https://aclanthology.org/2021.wmt-1.112\n",
    " [2]: https://huggingface.co/roberta-base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "a4396aea-bed0-4711-908a-a2b9522b91f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make decontextualized-similarity-matrix-roberta-base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "7d8f39d0-2c7e-4690-ab6c-12ce31609bf0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 254M\n",
      "-rw-r--r-- 1 1008 1011 720K May 30 11:05 'alpha=0.0'\n",
      "-rw-r--r-- 1 1008 1011  26M May 30 11:05 'alpha=0.1'\n",
      "-rw-r--r-- 1 1008 1011  26M May 30 11:05 'alpha=0.2'\n",
      "-rw-r--r-- 1 1008 1011  26M May 30 11:05 'alpha=0.3'\n",
      "-rw-r--r-- 1 1008 1011  26M May 30 11:05 'alpha=0.4'\n",
      "-rw-r--r-- 1 1008 1011  26M May 30 11:05 'alpha=0.5'\n",
      "-rw-r--r-- 1 1008 1011  26M May 30 11:05 'alpha=0.6'\n",
      "-rw-r--r-- 1 1008 1011  26M May 30 11:05 'alpha=0.7'\n",
      "-rw-r--r-- 1 1008 1011  26M May 30 11:05 'alpha=0.8'\n",
      "-rw-r--r-- 1 1008 1011  26M May 30 11:05 'alpha=0.9'\n",
      "-rw-r--r-- 1 1008 1011  26M May 30 11:05 'alpha=1.0'\n"
     ]
    }
   ],
   "source": [
    "%ls -lh decontextualized-similarity-matrix-roberta-base/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31f60209-a8eb-45bc-9bfe-42e70db301bc",
   "metadata": {},
   "source": [
    "### Decontextualized tuned `roberta-base` embeddings\n",
    "\n",
    "Next, we will produce term similarity matrices based on semantic similarities using the [decontextualized word embeddings][1] of [the `roberta-base` model][2] fine-tuned so that it can represent math-specific tokens.\n",
    "\n",
    " [1]: https://aclanthology.org/2021.wmt-1.112\n",
    " [2]: https://huggingface.co/roberta-base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "c819f65c-0526-475d-a7d3-825296696184",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make decontextualized-similarity-matrix-tuned-roberta-base-text+latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "e59f631e-4df5-49b4-8319-e629c5293d03",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 380M\n",
      "-rw-r--r-- 1 1008 1011 1.3M May 30 11:05 'alpha=0.0'\n",
      "-rw-r--r-- 1 1008 1011  38M May 30 11:05 'alpha=0.1'\n",
      "-rw-r--r-- 1 1008 1011  38M May 30 11:05 'alpha=0.2'\n",
      "-rw-r--r-- 1 1008 1011  38M May 30 11:05 'alpha=0.3'\n",
      "-rw-r--r-- 1 1008 1011  38M May 30 11:05 'alpha=0.4'\n",
      "-rw-r--r-- 1 1008 1011  38M May 30 11:05 'alpha=0.5'\n",
      "-rw-r--r-- 1 1008 1011  38M May 30 11:05 'alpha=0.6'\n",
      "-rw-r--r-- 1 1008 1011  38M May 30 11:05 'alpha=0.7'\n",
      "-rw-r--r-- 1 1008 1011  38M May 30 11:05 'alpha=0.8'\n",
      "-rw-r--r-- 1 1008 1011  38M May 30 11:05 'alpha=0.9'\n",
      "-rw-r--r-- 1 1008 1011  38M May 30 11:05 'alpha=1.0'\n"
     ]
    }
   ],
   "source": [
    "%ls -lh decontextualized-similarity-matrix-tuned-roberta-base-text+latex/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}