08-produce-arqmath-runs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "11cd3da2-44e7-4639-ab6b-a0f407130ef8",
   "metadata": {},
   "source": [
    "# Produce ARQMath runs\n",
    "\n",
    "In this notebook, we will runs on the ARQMath-1, ARQMath-2, and ARQMath-3 topics to be submitted to [the ARQMath-3 competition][1].\n",
    "\n",
    " [1]: https://www.cs.rit.edu/~dprl/ARQMath/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5ce20b63-7975-4337-b853-5b5bddcde3b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "docker.apollo.fi.muni.cz\n"
     ]
    }
   ],
   "source": [
    "! hostname"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "742f635e-9740-4e56-bfd4-506bf47dd72d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! pip install .[system]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c5c33c2-864b-4ae7-9a0b-8bec2e1ff101",
   "metadata": {},
   "source": [
    "## Joint soft vector space models\n",
    "\n",
    "First, we will produce runs using soft vector space models that jointly model both text and math. Information retrieval systems based on joint soft vector space models allow users to request math information using natural language and vise versa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f873786a-5320-43f0-a6ad-e06275f8c1a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "from pandas import DataFrame\n",
    "\n",
    "def evaluate_joint_run(basename: str) -> DataFrame:\n",
    "    with open(f'submission/{basename}.alpha_and_gamma', 'rt') as f:\n",
    "        alpha_and_gamma = json.load(f)\n",
    "    if 'alpha' in alpha_and_gamma or 'gamma' in alpha_and_gamma:\n",
    "        raise ValueError(f'Joint system from run {basename} is not yet optimized')\n",
    "\n",
    "    alpha = alpha_and_gamma['best_alpha']\n",
    "    gamma = alpha_and_gamma['best_gamma']\n",
    "    \n",
    "    with open(f'submission/{basename}.ndcg_score', 'rt') as f:\n",
    "        ndcg = f.read()\n",
    "\n",
    "    ndcg, *_ = ndcg.split(', ')\n",
    "    ndcg = float(ndcg)\n",
    "\n",
    "    formatters = {\"alpha\": lambda alpha: f'{alpha:.1f}',\n",
    "                  \"gamma\": lambda gamma: f'{gamma:g}',\n",
    "                  \"ndcg\": lambda ndcg: f'{alpha:.3f}'}\n",
    "\n",
    "    rows = 'ARQMath-3',\n",
    "    columns = 'α', 'γ', \"NDCG'\"\n",
    "    data = [[alpha, gamma, ndcg]]\n",
    "\n",
    "    dataframe = DataFrame(data, columns=columns, index=rows)\n",
    "\n",
    "    return dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f39cbbdf-9a33-411b-9ec6-b4fe7bced347",
   "metadata": {},
   "source": [
    "### The text format with no term similarities (baseline)\n",
    "\n",
    "As our baseline, we will use a vector space model that uses just text and does not model any term similarities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d12e23d9-3c5a-4c1d-a464-ee5b41d9e001",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make submission/SCM-task1-baseline_joint_text-text-auto-X.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f4a42965-ddba-4649-af48-36277ce33cd2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>α</th>\n",
       "      <th>γ</th>\n",
       "      <th>NDCG'</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ARQMath-3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.235</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             α  γ  NDCG'\n",
       "ARQMath-3  0.0  2  0.235"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_joint_run('SCM-task1-baseline_joint_text-text-auto-X')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e24b726c-cadc-4896-9230-90c2a9c61b29",
   "metadata": {},
   "source": [
    "### The text + LaTeX format with no term similarities (baseline)\n",
    "\n",
    "As another baseline, we will use a joint vector space model that uses text and LaTeX separated by special `[MATH]` and `[/MATH]` tokens, and does not model any term similarities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "91e1c8de-c3d6-465f-a79e-8b6895dbddcd",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make submission/SCM-task1-baseline_joint_text+latex-both-auto-X.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "75725cc3-5c3e-4bb1-b669-50073b1213c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>α</th>\n",
       "      <th>γ</th>\n",
       "      <th>NDCG'</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ARQMath-3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.224</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             α  γ  NDCG'\n",
       "ARQMath-3  0.0  3  0.224"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_joint_run('SCM-task1-baseline_joint_text+latex-both-auto-X')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24d1f3d0-219a-4cf6-b090-e14069605959",
   "metadata": {},
   "source": [
    "### The text + LaTeX format with non-positional `word2vec` embeddings\n",
    "\n",
    "As an alternative run, we will use a joint soft vector space model that uses text and LaTeX separated by special `[MATH]` and `[/MATH]` tokens, and models term similarities based on semantic similarities using `word2vec` models without [positional weighting][1].\n",
    "\n",
    " [1]: https://github.com/MIR-MU/pine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "db512183-b602-4196-b2c2-34652cf81859",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make submission/SCM-task1-joint_word2vec-both-auto-A.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "628fde62-cf6f-413d-91f7-f9f724ecaa04",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>α</th>\n",
       "      <th>γ</th>\n",
       "      <th>NDCG'</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ARQMath-3</th>\n",
       "      <td>0.6</td>\n",
       "      <td>5</td>\n",
       "      <td>0.251</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             α  γ  NDCG'\n",
       "ARQMath-3  0.6  5  0.251"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_joint_run('SCM-task1-joint_word2vec-both-auto-A')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "502ba237-94d3-4923-9fe4-25394e12e7c0",
   "metadata": {},
   "source": [
    "### The text + LaTeX format with positional `word2vec` embeddings\n",
    "\n",
    "As another alternative run, we will use a joint soft vector space model that uses text and LaTeX separated by special `[MATH]` and `[/MATH]` tokens, and models term similarities based on semantic similarities using `word2vec` models with [positional weighting][1].\n",
    "\n",
    " [1]: https://github.com/MIR-MU/pine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "181b2b9f-6fc5-4355-ab96-3dbb30d29b38",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make submission/SCM-task1-joint_positional_word2vec-both-auto-A.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "69462399-d1f3-463a-870e-b80400519253",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>α</th>\n",
       "      <th>γ</th>\n",
       "      <th>NDCG'</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ARQMath-3</th>\n",
       "      <td>0.7</td>\n",
       "      <td>5</td>\n",
       "      <td>0.249</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             α  γ  NDCG'\n",
       "ARQMath-3  0.7  5  0.249"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_joint_run('SCM-task1-joint_positional_word2vec-both-auto-A')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27ee922e-00aa-41eb-b655-609c7f8f29b0",
   "metadata": {},
   "source": [
    "### The text format with decontextualized `roberta-base` embeddings\n",
    "\n",
    "As another alternative run, we will use a joint soft vector space model that uses just text and models term similarities based on semantic similarities using the [decontextualized word embeddings][1] of [the `roberta-base` model][2].\n",
    "\n",
    " [1]: https://aclanthology.org/2021.wmt-1.112\n",
    " [2]: https://huggingface.co/roberta-base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "536b8c7c-1932-41b8-8915-fff3a5d4d8a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make submission/SCM-task1-joint_roberta_base-text-auto-A.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b13948d6-bcf2-4c59-9f6f-c9de6f1c7a32",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>α</th>\n",
       "      <th>γ</th>\n",
       "      <th>NDCG'</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ARQMath-3</th>\n",
       "      <td>0.6</td>\n",
       "      <td>2</td>\n",
       "      <td>0.247</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             α  γ  NDCG'\n",
       "ARQMath-3  0.6  2  0.247"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_joint_run('SCM-task1-joint_roberta_base-text-auto-A')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1810ce1-d765-4792-a950-028634e22772",
   "metadata": {},
   "source": [
    "### The text + LaTeX format with decontextualized tuned `roberta-base` embeddings\n",
    "\n",
    "As another alternative run, we will use a joint soft vector space model that uses text and LaTeX separated by special `[MATH]` and `[/MATH]` tokens, and models term similarities based on semantic similarities using the [decontextualized word embeddings][1] of [the `roberta-base` model][2] fine-tuned so that it can represent math-specific tokens.\n",
    "\n",
    " [1]: https://aclanthology.org/2021.wmt-1.112\n",
    " [2]: https://huggingface.co/roberta-base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "124d80d3-a35c-454e-a6a3-534ffe8feb36",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make submission/SCM-task1-joint_tuned_roberta_base-both-auto-A.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "78c5ae42-ced6-42bb-8bb7-100b2aa76b46",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>α</th>\n",
       "      <th>γ</th>\n",
       "      <th>NDCG'</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ARQMath-3</th>\n",
       "      <td>0.6</td>\n",
       "      <td>4</td>\n",
       "      <td>0.249</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             α  γ  NDCG'\n",
       "ARQMath-3  0.6  4  0.249"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_joint_run('SCM-task1-joint_tuned_roberta_base-both-auto-A')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c31c44da-607b-4795-a0fc-419f522b554d",
   "metadata": {},
   "source": [
    "## Interpolated soft vector space models\n",
    "\n",
    "Secondly, we will produce runs using soft vector space models that model text and math separately and produce the final score of a document by interpolating scores for text and math. Interpolated soft vector space models are better theoretically motivated and more modular than joint vector space models, but they cannot model the similarities between text and math-specific tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2586a41f-d4bc-455c-8478-00791be1954a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "from pandas import DataFrame\n",
    "\n",
    "def evaluate_interpolated_run(basename: str) -> DataFrame:\n",
    "    with open(f'submission/{basename}.first_alpha_and_gamma', 'rt') as f:\n",
    "        first_alpha_and_gamma = json.load(f)\n",
    "    if 'alpha' in first_alpha_and_gamma or 'gamma' in first_alpha_and_gamma:\n",
    "        raise ValueError(f'First system from run {basename} is not yet optimized')\n",
    "\n",
    "    first_alpha = first_alpha_and_gamma['best_alpha']\n",
    "    first_gamma = first_alpha_and_gamma['best_gamma']\n",
    "\n",
    "    with open(f'submission/{basename}.second_alpha_and_gamma', 'rt') as f:\n",
    "        second_alpha_and_gamma = json.load(f)\n",
    "    if 'alpha' in second_alpha_and_gamma or 'gamma' in second_alpha_and_gamma:\n",
    "        raise ValueError(f'Second system from run {basename} is not yet optimized')\n",
    "\n",
    "    second_alpha = second_alpha_and_gamma['best_alpha']\n",
    "    second_gamma = second_alpha_and_gamma['best_gamma']\n",
    "\n",
    "    with open(f'submission/{basename}.beta', 'rt') as f:\n",
    "        _beta = json.load(f)\n",
    "    if 'beta' in _beta:\n",
    "        raise ValueError(f'Interpolated system from run {basename} is not yet optimized')\n",
    "    \n",
    "    beta = _beta['best_beta']\n",
    "    \n",
    "    with open(f'submission/{basename}.ndcg_score', 'rt') as f:\n",
    "        ndcg = f.read()\n",
    "\n",
    "    ndcg, *_ = ndcg.split(', ')\n",
    "    ndcg = float(ndcg)\n",
    "        \n",
    "    formatters = {\"first_alpha\": lambda alpha: f'{alpha:.1f}',\n",
    "                  \"first_gamma\": lambda gamma: f'{gamma:g}',\n",
    "                  \"second_alpha\": lambda alpha: f'{alpha:.1f}',\n",
    "                  \"second_gamma\": lambda gamma: f'{gamma:g}',\n",
    "                  \"beta\": lambda beta: f'{beta:.1f}',\n",
    "                  \"ndcg\": lambda ndcg: f'{alpha:.3f}'}\n",
    "\n",
    "    rows = 'ARQMath-3',\n",
    "    columns = 'α₁', 'γ₁', 'α₂', 'γ₂', 'β', \"NDCG'\"\n",
    "    data = [[first_alpha, first_gamma, second_alpha, second_gamma, beta, ndcg]]\n",
    "\n",
    "    dataframe = DataFrame(data, columns=columns, index=rows)\n",
    "\n",
    "    return dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5f64f89-cb77-416f-8a76-9aa8f98a2010",
   "metadata": {},
   "source": [
    "### The text + LaTeX format with no term similarities (baseline)\n",
    "\n",
    "As a baseline, we will use interpolated vector space models that use text and LaTeX separated by special `[MATH]` and `[/MATH]` tokens, and does not model any term similarities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "7e8fe15c-ffc5-49c0-8027-f32eb9a104f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make submission/SCM-task1-baseline_interpolated_text+latex-both-auto-X.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8cf3d73e-30fa-4f4e-b7fc-4fe4c0994bb8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>α₁</th>\n",
       "      <th>γ₁</th>\n",
       "      <th>α₂</th>\n",
       "      <th>γ₂</th>\n",
       "      <th>β</th>\n",
       "      <th>NDCG'</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ARQMath-3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.257</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            α₁  γ₁   α₂  γ₂    β  NDCG'\n",
       "ARQMath-3  0.0   2  0.0   5  0.6  0.257"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_interpolated_run('SCM-task1-baseline_interpolated_text+latex-both-auto-X')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d72091f0-6c8a-45e1-8719-90895f93d716",
   "metadata": {},
   "source": [
    "### The text + Tangent-L format with no term similarities (baseline)\n",
    "\n",
    "As another baseline, we will use interpolated vector space models that use text and the format used by [the Tangent-L search engine from UWaterloo][1], and does not model any term similarities.\n",
    "\n",
    " [1]: http://ceur-ws.org/Vol-2936/paper-05.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "36d5478b-b250-4c95-9076-f19aa84eb549",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make submission/SCM-task1-baseline_interpolated_text+tangentl-both-auto-X.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "357d7396-928b-4788-a90d-5b4fb97ad69f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>α₁</th>\n",
       "      <th>γ₁</th>\n",
       "      <th>α₂</th>\n",
       "      <th>γ₂</th>\n",
       "      <th>β</th>\n",
       "      <th>NDCG'</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ARQMath-3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.349</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            α₁  γ₁   α₂  γ₂    β  NDCG'\n",
       "ARQMath-3  0.0   2  0.0   4  0.6  0.349"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_interpolated_run('SCM-task1-baseline_interpolated_text+tangentl-both-auto-X')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb4c8a88-134e-4c69-a027-1f927f3156c0",
   "metadata": {},
   "source": [
    "### The text + LaTeX format with non-positional `word2vec` embeddings\n",
    "\n",
    "As an alternative run, we will use interpolated soft vector space models that use text and LaTeX separated by special `[MATH]` and `[/MATH]` tokens. The LaTeX soft vector space model uses term similarities based on semantic similarities using `word2vec` models without [positional weighting][1].\n",
    "\n",
    " [1]: https://github.com/MIR-MU/pine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "7f7a14a8-4687-4cab-bf3c-29778a6675f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make submission/SCM-task1-interpolated_word2vec_text+latex-both-auto-A.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "6e1681c5-bff7-46f7-90b5-09b748ff601b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>α₁</th>\n",
       "      <th>γ₁</th>\n",
       "      <th>α₂</th>\n",
       "      <th>γ₂</th>\n",
       "      <th>β</th>\n",
       "      <th>NDCG'</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ARQMath-3</th>\n",
       "      <td>0.6</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.288</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            α₁  γ₁   α₂  γ₂    β  NDCG'\n",
       "ARQMath-3  0.6   2  1.0   5  0.6  0.288"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_interpolated_run('SCM-task1-interpolated_word2vec_text+latex-both-auto-A')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18820249-4ad0-461f-a96f-31a2052967c7",
   "metadata": {},
   "source": [
    "### The text + LaTeX format with positional `word2vec` embeddings\n",
    "\n",
    "As another alternative run, we will use interpolated soft vector space models that use text and LaTeX separated by special `[MATH]` and `[/MATH]` tokens. The LaTeX model uses term similarities based on semantic similarities using `word2vec` models with [positional weighting][1].\n",
    "\n",
    " [1]: https://github.com/MIR-MU/pine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "876343b5-f185-45ac-b21b-a036544922c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make submission/SCM-task1-interpolated_positional_word2vec_text+latex-both-auto-A.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "68ae2585-afca-4c45-a8b2-1f89b4b1328e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>α₁</th>\n",
       "      <th>γ₁</th>\n",
       "      <th>α₂</th>\n",
       "      <th>γ₂</th>\n",
       "      <th>β</th>\n",
       "      <th>NDCG'</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ARQMath-3</th>\n",
       "      <td>0.7</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.288</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            α₁  γ₁   α₂  γ₂    β  NDCG'\n",
       "ARQMath-3  0.7   2  1.0   5  0.6  0.288"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_interpolated_run('SCM-task1-interpolated_positional_word2vec_text+latex-both-auto-A')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d34ba83f-590a-4c1c-9711-765221f48a79",
   "metadata": {},
   "source": [
    "### The text + Tangent-L format with non-positional `word2vec` embeddings\n",
    "\n",
    "As another alternative run, we will use interpolated soft vector space models that use text and LaTeX separated by special `[MATH]` and `[/MATH]` tokens. The LaTeX soft vector space model uses term similarities based on semantic similarities using `word2vec` models without [positional weighting][1].\n",
    "\n",
    " [1]: https://github.com/MIR-MU/pine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "038686a0-ace0-44c3-8e1a-ed685a645084",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make submission/SCM-task1-interpolated_word2vec_text+tangentl-both-auto-A.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "5afeb9e1-f778-460d-b5a0-19faf11269e6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>α₁</th>\n",
       "      <th>γ₁</th>\n",
       "      <th>α₂</th>\n",
       "      <th>γ₂</th>\n",
       "      <th>β</th>\n",
       "      <th>NDCG'</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ARQMath-3</th>\n",
       "      <td>0.6</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5</td>\n",
       "      <td>0.7</td>\n",
       "      <td>0.351</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            α₁  γ₁   α₂  γ₂    β  NDCG'\n",
       "ARQMath-3  0.6   2  0.0   5  0.7  0.351"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_interpolated_run('SCM-task1-interpolated_word2vec_text+tangentl-both-auto-A')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb4aa6fa-b3e1-46b9-8b4c-3a445d23f190",
   "metadata": {},
   "source": [
    "### The text + Tangent-L format with positional `word2vec` embeddings\n",
    "\n",
    "As our primary run, we will use interpolated soft vector space models that use text and LaTeX separated by special `[MATH]` and `[/MATH]` tokens. The LaTeX model uses term similarities based on semantic similarities using `word2vec` models with [positional weighting][1].\n",
    "\n",
    " [1]: https://github.com/MIR-MU/pine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "29c7fe18-acac-4a8f-839d-a5dbb9d4b7f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make submission/SCM-task1-interpolated_positional_word2vec_text+tangentl-both-auto-P.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "7d45f62b-173b-4b99-8a21-c8310862ac5f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>α₁</th>\n",
       "      <th>γ₁</th>\n",
       "      <th>α₂</th>\n",
       "      <th>γ₂</th>\n",
       "      <th>β</th>\n",
       "      <th>NDCG'</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ARQMath-3</th>\n",
       "      <td>0.7</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5</td>\n",
       "      <td>0.7</td>\n",
       "      <td>0.355</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            α₁  γ₁   α₂  γ₂    β  NDCG'\n",
       "ARQMath-3  0.7   2  0.0   5  0.7  0.355"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_interpolated_run('SCM-task1-interpolated_positional_word2vec_text+tangentl-both-auto-P')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}