Binary_Inference_Dictionaries_Electoral_NLP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b7f3eace-4792-484e-b45b-25b09b852a27",
   "metadata": {},
   "source": [
    "# **B**inary **I**nference **D**ictionaries for **E**lectoral **N**LP"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96e7885e-2c83-4623-b4a4-3ed3d3f8954e",
   "metadata": {},
   "source": [
    "![BIDEN](BIDEN.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffcbf072-b7ae-4d30-b54d-e2dff523afc5",
   "metadata": {},
   "source": [
    "_2023-10-01_\n",
    "\n",
    "**By Matt Hodges**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e11e2ae-8796-431a-880b-2fdf7f34905d",
   "metadata": {},
   "source": [
    "**BIDEN**: **B**inary **I**nference **D**ictionaries for **E**lectoral **N**LP\n",
    "\n",
    "This Notebook demonstrates a compression-based binary classification technique that is fast at both training and inference on common CPU hardware in Python. It is largely built on the strategies presented by [FTCC](https://github.com/cyrilou242/ftcc), which in turn, was a reaction to [Low-Resource Text Classification: A Parameter-Free Classification Method with Compressors](https://github.com/bazingagin/npc_gzip) (the gzip method). Like FTCC, **BIDEN** is built atop of [Zstandard](https://facebook.github.io/zstd/) (Zstd), which leverages [dictionary compression](https://facebook.github.io/zstd/#small-data). Zstd dictionary compression seeds a compressor with sample data, so that it can efficiently compress _small data_ (~1 KB) of similar composition. Seeding the compressor dictionaries acts as our \"training\" method for the model.\n",
    "\n",
    "The **BIDEN** model was trained on the [ElectionEmails 2020](https://electionemails2020.org) data set — a database of over 900,000 political campaign emails from the 2020 US election cycle. **In compliance with the data set's [terms](https://electionemails2020.org/downloads/corpus_documentation_v1.0.pdf), the training data is NOT provided with this repository.** If you would like to train the **BIDEN** model yourself, you can [request a copy of the data for free](https://docs.google.com/forms/d/e/1FAIpQLSdcgjZo-D1nNON4d90H2j0VLtTdxiHK6Y8HPJSpdRu4w5YILw/viewform). The **BIDEN** model was trained on `corpus_v1.0`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "505dfb9a-0b46-4dfb-89c0-d19e3aeb117d",
   "metadata": {},
   "source": [
    "## Training and Classification\n",
    "\n",
    "Both training and inference for **BIDEN** are fast and simple.\n",
    "\n",
    "The model consists of two Zstd compressors, one optimized for Democratic emails and one optimzed for Republican emails. Each is built upon a compression dictionary. Each compression dictionary is seeded with training sample emails from its respective party.\n",
    "\n",
    "Classification (inference) is achieved by compressing a test sample with both the Democratic and Republican compressors. Whichever compressor achieves a higher compression ratio on the test sample text is considered the inferred label."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "407d88bc-f6da-4b9a-a1cd-256ad6ed724f",
   "metadata": {},
   "source": [
    "## Cleaning the Training Data\n",
    "\n",
    "The ElectionEmails 2020 data set is a CSV. The model consideres [two columns](https://electionemails2020.org/downloads/corpus_documentation_v1.0.pdf): `party_affiliation` and `body_text`. **BIDEN** is only concerned with binary classification for Democratic and Republican labeling.\n",
    "\n",
    "The two requirements defined in `requirements.txt` are [Pandas](https://pypi.org/project/pandas/) and [zstandard](https://pypi.org/project/zstandard/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b96e8d40-9eb2-4162-a920-9ad88d63d3ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install -r requirements.txt --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa210e88-22ef-438e-9fcb-77adcbb46cad",
   "metadata": {},
   "source": [
    "Start by reading in the data. Since the model is only working with two columns, drop any record that doesn't contain both. Also filter the data to only consider Democratic or Republican emails for the binary classificaiton.\n",
    "\n",
    "**Note**: this assumes you have the ElectionEmails 2020 data saved at the relative path `data/corpus_v1.0.csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a8127153-8f5a-460c-a5d5-da2fcc7c8d8c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "D Samples: 127194\n",
      "R Samples: 36788\n"
     ]
    }
   ],
   "source": [
    "from enum import Enum\n",
    "\n",
    "import pandas as pd\n",
    "import zstandard\n",
    "\n",
    "fields = {\n",
    "    'body_text': str,\n",
    "    'party_affiliation': str,\n",
    "}\n",
    "\n",
    "df = pd.read_csv(\n",
    "    'data/corpus_v1.0.csv',\n",
    "    sep=',',\n",
    "    usecols=list(fields.keys()),\n",
    "    dtype=fields,\n",
    ")\n",
    "\n",
    "df.drop_duplicates(inplace=True)\n",
    "\n",
    "d_df = df[df.party_affiliation == \"Democratic Party\"].dropna()\n",
    "r_df = df[df.party_affiliation == \"Republican Party\"].dropna()\n",
    "\n",
    "print(f'D Samples: {len(d_df.index)}')\n",
    "print(f'R Samples: {len(r_df.index)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6121da6-bbbb-4b7d-a2cb-c053fe0b8b72",
   "metadata": {},
   "source": [
    "There are significantly more Democratic samples than Republican samples, so take a random subset of the former."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "54b8fe57-dae5-47dd-af3c-aa7c9d28d853",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "D Samples: 36788\n",
      "R Samples: 36788\n"
     ]
    }
   ],
   "source": [
    "max_data = min(len(d_df.index), len(r_df.index))\n",
    "d_df = d_df.sample(\n",
    "    n=max_data,\n",
    "    random_state=9001  # random seed set for reproducibility \n",
    ").reset_index(drop=True)\n",
    "\n",
    "r_df = r_df.sample(\n",
    "    n=max_data,\n",
    "    random_state=9001  # random seed set for reproducibility \n",
    ").reset_index(drop=True)\n",
    "\n",
    "print(f'D Samples: {len(d_df.index)}')\n",
    "print(f'R Samples: {len(r_df.index)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43594767-784a-4972-bca6-699a324e32a9",
   "metadata": {},
   "source": [
    "Now divide the data into training and test subsets, at an 80/20 split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7f3c0940-4013-4e03-b361-bdf5886f9954",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Democratic Training Samples: 29430\n",
      "Democratic Test Samples: 7358\n",
      "Republican Training Samples: 29430\n",
      "Republican Test Samples: 7358\n"
     ]
    }
   ],
   "source": [
    "d_train_df = d_df.sample(frac=0.8, random_state=9001)  # random seed set for reproducibility \n",
    "d_test_df = d_df.drop(d_train_df.index)\n",
    "\n",
    "r_train_df = r_df.sample(frac=0.8, random_state=9001)  # random seed set for reproducibility \n",
    "r_test_df = r_df.drop(r_train_df.index)\n",
    "\n",
    "print(f'Democratic Training Samples: {len(d_train_df.index)}')\n",
    "print(f'Democratic Test Samples: {len(d_test_df.index)}')\n",
    "print(f'Republican Training Samples: {len(r_train_df.index)}')\n",
    "print(f'Republican Test Samples: {len(r_test_df.index)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cb3506c-859c-4887-871f-49717f7f7672",
   "metadata": {},
   "source": [
    "## The BIDEN model\n",
    "\n",
    "The model consistes of two core methods: `train()` and `classify()` :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "53a77efa-3809-4bfe-9afb-42ca058943fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "class BIDEN():\n",
    "    \"\"\"\n",
    "    Binary Inference Dictionaries for Electoral NLP (BIDEN)\n",
    "\n",
    "    This class allows you to train a model for classifying political content into\n",
    "    Democratic or Republican categories based on compression ratios.\n",
    "\n",
    "    Attributes:\n",
    "        Classification (enum): An enumeration of political classifications (DEMOCRATIC, REPUBLICAN).\n",
    "    \"\"\"\n",
    "    class Classification(Enum):\n",
    "        \"\"\"\n",
    "        Enumeration of political classifications.\n",
    "\n",
    "        Attributes:\n",
    "            DEMOCRATIC (int): Represents Democratic political content.\n",
    "            REPUBLICAN (int): Represents Republican political content.\n",
    "        \"\"\"\n",
    "        DEMOCRATIC = 1\n",
    "        REPUBLICAN = 2\n",
    "        \n",
    "    def __init__(self, encoding: str = 'utf-8'):\n",
    "        \"\"\"\n",
    "        Initialize the BIDEN model.\n",
    "\n",
    "        This constructor initializes the BIDEN model with empty compressors.\n",
    "\n",
    "        Args:\n",
    "            encoding (str, optional): The character encoding of the input data. Defaults to 'utf-8'.\n",
    "            \n",
    "        Returns:\n",
    "            BIDEN: An instance of the BIDEN class.\n",
    "        \"\"\"\n",
    "        self.d_compressor = None\n",
    "        self.d_compressor = None\n",
    "        self.encoding = encoding\n",
    "\n",
    "    @property\n",
    "    def trained(self) -> bool:\n",
    "        \"\"\"\n",
    "        Check if the BIDEN model is trained.\n",
    "\n",
    "        Returns:\n",
    "            bool: True if both Democratic and Republican compressors are trained, False otherwise.\n",
    "        \"\"\"\n",
    "        return bool(self.d_compressor and self.r_compressor)\n",
    "\n",
    "    def train(self,\n",
    "              d_training_data: str,\n",
    "              r_training_data: str,\n",
    "              compression_level: int = 15,\n",
    "             ) -> bool:\n",
    "        \"\"\"\n",
    "        Train the BIDEN model.\n",
    "\n",
    "        Args:\n",
    "            d_training_data (str): Democratic training data.\n",
    "            r_training_data (str): Republican training data.\n",
    "            compression_level (int, optional): The compression level. Defaults to 15.\n",
    "\n",
    "        Returns:\n",
    "            bool: True if training is successful, False otherwise.\n",
    "        \"\"\"        \n",
    "        d_dictionary = zstandard.ZstdCompressionDict(\n",
    "            d_training_data.encode(self.encoding),\n",
    "            dict_type=zstandard.DICT_TYPE_RAWCONTENT\n",
    "        )\n",
    "        d_dictionary.precompute_compress(level=compression_level)\n",
    "        self.d_compressor = zstandard.ZstdCompressor(dict_data=d_dictionary)\n",
    "\n",
    "        r_dictionary = zstandard.ZstdCompressionDict(\n",
    "            r_training_data.encode(self.encoding),\n",
    "            dict_type=zstandard.DICT_TYPE_RAWCONTENT\n",
    "        )\n",
    "        r_dictionary.precompute_compress(level=compression_level)\n",
    "        self.r_compressor = zstandard.ZstdCompressor(dict_data=r_dictionary)\n",
    "\n",
    "        return self.trained\n",
    "\n",
    "    def classify(self, sample: str) -> Classification:\n",
    "        \"\"\"\n",
    "        Classify a sample based on compression ratios.\n",
    "\n",
    "        Args:\n",
    "            sample (str): The sample text to classify.\n",
    "\n",
    "        Returns:\n",
    "            Classification: The classification (DEMOCRATIC or REPUBLICAN).\n",
    "        \n",
    "        Raises:\n",
    "            RuntimeError: If the model is not trained.\n",
    "        \"\"\"\n",
    "        if not self.trained:\n",
    "            raise RuntimeError(\"Attempted to classify with a model that is not yet trained.\")\n",
    "        \n",
    "        encoded_sample = sample.encode(self.encoding)\n",
    "        original_length = len(encoded_sample)\n",
    "        d_compressed_length = len(self.d_compressor.compress(encoded_sample))\n",
    "        d_ratio = d_compressed_length / original_length\n",
    "        r_compressed_length = len(self.r_compressor.compress(encoded_sample))\n",
    "        r_ratio = r_compressed_length / original_length\n",
    "\n",
    "        if r_ratio < d_ratio:\n",
    "            return BIDEN.Classification.REPUBLICAN\n",
    "\n",
    "        return BIDEN.Classification.DEMOCRATIC"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79db5fe1-54f8-48a4-a283-1155ab4f18f2",
   "metadata": {},
   "source": [
    "## Train the Model\n",
    "\n",
    "To train the model, we pass the entirety of the Democratic and Republican texts to construct [prefix dictionaries](https://python-zstandard.readthedocs.io/en/latest/dictionaries.html). Prefix dictionaries allow compression operations to reference raw data within the [dictionary](https://python-zstandard.readthedocs.io/en/latest/concepts.html#dictionaries). Once we have two compressors instantiated and pre-seeded with our training data, the model is trained. This is _fast_. On my 2.6 GHz 6-Core Intel Core i7, this takes roughly **30 seconds**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cfa1300f-0ca7-4030-942f-7ea780e6db4e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d_combined_text = '\\n'.join(d_train_df.body_text)\n",
    "r_combined_text = '\\n'.join(r_train_df.body_text)\n",
    "\n",
    "model = BIDEN()\n",
    "model.train(d_combined_text, r_combined_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3747d3d3-0ef3-4b79-9c3c-e980ed51f917",
   "metadata": {},
   "source": [
    "## Classification\n",
    "\n",
    "Now, we can classify our test data. We could loop through each set, but let's combine and shuffle them together first, and loop in one go. We'll also convert the party affiliation strings `'Democratic Party'`, and `'Republican Party'` into our model's enum values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8179375a-26b8-46f9-8de0-8f44eef4b141",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification Success Rate: 98.9%\n"
     ]
    }
   ],
   "source": [
    "combined_test_df = pd.concat(\n",
    "    [d_test_df, r_test_df],\n",
    "    ignore_index=True,\n",
    "    sort=False\n",
    ").sample(\n",
    "    frac=1,\n",
    "    random_state=9001,  # random seed set for reproducibility\n",
    ").reset_index(drop=True)\n",
    "\n",
    "combined_test_df['party_affiliation'] = combined_test_df['party_affiliation'].replace(\n",
    "    to_replace=['Democratic Party', 'Republican Party'],\n",
    "    value=[BIDEN.Classification.DEMOCRATIC, BIDEN.Classification.REPUBLICAN]\n",
    ")\n",
    "\n",
    "num_correct = 0\n",
    "for row in combined_test_df.itertuples():\n",
    "    actual_label = row.party_affiliation\n",
    "    inferred_label = model.classify(row.body_text)\n",
    "\n",
    "    if inferred_label == actual_label:\n",
    "        num_correct += 1\n",
    "\n",
    "print(f'Classification Success Rate: {((num_correct / len(combined_test_df.index))*100):.1f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29ed9414-9676-4a7c-8355-74d00fbeb3ef",
   "metadata": {},
   "source": [
    "98.9% is a _shockingly_ high success rate for such a simple classification method!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78e3f3c1-01f3-4c55-8aee-edb36a999abb",
   "metadata": {},
   "source": [
    "## Another Email Data Set\n",
    "\n",
    "Let's see how it performs with emails from another data set. [Derek Willis](https://www.thescoop.org) maintains a [Datasette](https://datasette.io/) of [over 150,000 campaign emails](https://political-emails.herokuapp.com/emails). Let's grab 100 samples of each party from that collection, and see how they perform:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "95f0a04c-a454-44e9-b189-c45a9d17cb0c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DW Sample:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>body</th>\n",
       "      <th>party</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Hey Peter!!! You know our campaign is made by,...</td>\n",
       "      <td>Classification.DEMOCRATIC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://www.jahanahayes.com/ [https://www.jaha...</td>\n",
       "      <td>Classification.DEMOCRATIC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hi there, it’s storytime: I developed a keen n...</td>\n",
       "      <td>Classification.DEMOCRATIC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>We’re contacting a select group of patriots fo...</td>\n",
       "      <td>Classification.REPUBLICAN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Peter – You need to watch this. CLICK HERE OR ...</td>\n",
       "      <td>Classification.REPUBLICAN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Peter, you may have seen me asking for your ol...</td>\n",
       "      <td>Classification.DEMOCRATIC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Do you want an Official Mugshot Mug? (24-hour ...</td>\n",
       "      <td>Classification.REPUBLICAN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>https://secure.actblue.com/donate/cdp-footer?r...</td>\n",
       "      <td>Classification.DEMOCRATIC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[ https://act.katieporter.com/go/14559?t=1001&amp;...</td>\n",
       "      <td>Classification.DEMOCRATIC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>I hope I made you proud fighting for you at th...</td>\n",
       "      <td>Classification.REPUBLICAN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                body  \\\n",
       "0  Hey Peter!!! You know our campaign is made by,...   \n",
       "1  https://www.jahanahayes.com/ [https://www.jaha...   \n",
       "2  Hi there, it’s storytime: I developed a keen n...   \n",
       "3  We’re contacting a select group of patriots fo...   \n",
       "4  Peter – You need to watch this. CLICK HERE OR ...   \n",
       "5  Peter, you may have seen me asking for your ol...   \n",
       "6  Do you want an Official Mugshot Mug? (24-hour ...   \n",
       "7  https://secure.actblue.com/donate/cdp-footer?r...   \n",
       "8  [ https://act.katieporter.com/go/14559?t=1001&...   \n",
       "9  I hope I made you proud fighting for you at th...   \n",
       "\n",
       "                       party  \n",
       "0  Classification.DEMOCRATIC  \n",
       "1  Classification.DEMOCRATIC  \n",
       "2  Classification.DEMOCRATIC  \n",
       "3  Classification.REPUBLICAN  \n",
       "4  Classification.REPUBLICAN  \n",
       "5  Classification.DEMOCRATIC  \n",
       "6  Classification.REPUBLICAN  \n",
       "7  Classification.DEMOCRATIC  \n",
       "8  Classification.DEMOCRATIC  \n",
       "9  Classification.REPUBLICAN  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "base_url = 'https://political-emails.herokuapp.com/emails.csv?sql='\n",
    "d_query = 'select body, party from emails where party = \"D\" limit 100;'\n",
    "r_query = 'select body, party from emails where party = \"R\" limit 100;'\n",
    "\n",
    "map = {\n",
    "    ' ': '+',\n",
    "    ',': '%2C',\n",
    "    '=': '%3D',\n",
    "    '\"': '%22',\n",
    "    ';': '%3B',\n",
    "}\n",
    "\n",
    "for symbol, code in map.items():\n",
    "    d_query = d_query.replace(symbol, code)\n",
    "    r_query = r_query.replace(symbol, code)\n",
    "\n",
    "d_url = base_url + d_query\n",
    "r_url = base_url + r_query\n",
    "\n",
    "d_dw_df = pd.read_csv(d_url)\n",
    "r_dw_df = pd.read_csv(r_url)\n",
    "\n",
    "combined_dw_df = pd.concat(\n",
    "    [d_dw_df, r_dw_df],\n",
    "    ignore_index=True,\n",
    "    sort=False\n",
    ").sample(\n",
    "    frac=1,\n",
    "    random_state=9001,  # random seed set for reproducibility\n",
    ").reset_index(drop=True)\n",
    "\n",
    "combined_dw_df['party'] = combined_dw_df['party'].replace(\n",
    "    to_replace=['D', 'R'],\n",
    "    value=[BIDEN.Classification.DEMOCRATIC, BIDEN.Classification.REPUBLICAN]\n",
    ")\n",
    "\n",
    "print('DW Sample:')\n",
    "combined_dw_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e8cc55d0-fba3-4a6a-ad2f-5e5db9da3016",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification Success Rate: 93.0%\n"
     ]
    }
   ],
   "source": [
    "num_correct = 0\n",
    "for row in combined_dw_df.itertuples():\n",
    "    actual_label = row.party\n",
    "    inferred_label = model.classify(row.body)\n",
    "\n",
    "    if inferred_label == actual_label:\n",
    "        num_correct += 1\n",
    "\n",
    "print(f'Classification Success Rate: {((num_correct / len(combined_dw_df.index))*100):.1f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f647cae-854d-45b8-a9d8-baaaadc32f8f",
   "metadata": {},
   "source": [
    "93% is still _quite good_ considering that all we're doing is compression!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbdba830-45a1-4d1a-b171-6457cbeb32fc",
   "metadata": {},
   "source": [
    "## Classifying Tweets\n",
    "\n",
    "The Twitter API is basically useless these days, so I scrolled the timelines of [Mike Pence](https://twitter.com/Mike_Pence) (R) and [Gavin Newsom](https://twitter.com/GavinNewsom) (D), and copy+paste'd 5 tweets from each. It's a tiny sample, and not really random, but it's neat to see how well it does:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5839d1f1-3b63-41da-a131-d594b0fb2b00",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Classification.REPUBLICAN: 2>,\n",
       " <Classification.REPUBLICAN: 2>,\n",
       " <Classification.REPUBLICAN: 2>,\n",
       " <Classification.REPUBLICAN: 2>,\n",
       " <Classification.REPUBLICAN: 2>]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pence_tweets = [\n",
    "    # https://twitter.com/Mike_Pence/status/1707882018258751915\n",
    "    \"Today, we applaud the Eighth Circuit's decision, \" \\\n",
    "    \"which is an important step in the fight to uphold and protect \" \\\n",
    "    \"the rights of parents and families in Linn-Mar. A family is the \" \\\n",
    "    \"single best unit to protect children, and we must continue to do \" \\\n",
    "    \"everything we can to empower parents over bureaucrats. The strength \" \\\n",
    "    \"of our nation is tied to the strength of our families, and we will \" \\\n",
    "    \"not stand idly by as the Radical Left attempts to indoctrinate our \" \\\n",
    "    \"children behind parents’ backs.\",\n",
    "    # https://twitter.com/Mike_Pence/status/1707472823269392643\n",
    "    \"The cause of Life is the calling of our time \" \\\n",
    "    \"and I couldn’t be more proud to have played a role in the administration \" \\\n",
    "    \"that appointed three of the justices that overturned Roe v. Wade and \" \\\n",
    "    \"returned the question of abortion to the states and the American people.\",\n",
    "    \"Republicans are facing a Time for Choosing. \" \\\n",
    "    # https://twitter.com/Mike_Pence/status/1707241587460186214\n",
    "    \"We have to choose whether or not we’re going to stay on the path \" \\\n",
    "    \"that has defined our movement since the days of Ronald Reagan and \" \\\n",
    "    \"through the Trump-Pence years or whether we’re going to follow \" \\\n",
    "    \"the siren song of populism unmoored to Conservative principles.\",\n",
    "    # https://twitter.com/Mike_Pence/status/1704132623617122719\n",
    "    \"I am for working Americans and free enterprise. These businesses \" \\\n",
    "    \"make their decisions, but at the end the of the day these businesses \" \\\n",
    "    \"are responding to the heavy hand of the green new deal agenda of \" \\\n",
    "    \"Joe Biden and the Democrats in Washington, DC.\",\n",
    "    # https://twitter.com/Mike_Pence/status/1703887286641873137\n",
    "    \"We were the first country to sanction Chinese leaders for building \" \\\n",
    "    \"concentration camps in Xinjiang and for undermining democracy in Hong Kong. \" \\\n",
    "    \"And we stood up to years of trade abuses, and imposed historic tariffs \" \\\n",
    "    \"to bring China to the negotiating table.\"\n",
    "]\n",
    "\n",
    "[model.classify(t) for t in pence_tweets]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9d56a846-ff42-4744-ad85-98842226edfa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Classification.DEMOCRATIC: 1>,\n",
       " <Classification.DEMOCRATIC: 1>,\n",
       " <Classification.DEMOCRATIC: 1>,\n",
       " <Classification.DEMOCRATIC: 1>,\n",
       " <Classification.REPUBLICAN: 2>]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "newsom_tweets = [\n",
    "    # https://twitter.com/GavinNewsom/status/1700615276667294035\n",
    "    \"When people ask why I am introducing a Constitutional Amendment \" \\\n",
    "    \"on gun safety this is why. Not only has Congress stalled for YEARS on passing \" \\\n",
    "    \"common sense reforms -- judges across the country are tearing down laws that \" \\\n",
    "    \"Americans overwhelmingly support. Laws that keep us SAFE and keep guns out of \" \\\n",
    "    \"the hands of dangerous criminals. We have to push back\",\n",
    "    # # https://twitter.com/GavinNewsom/status/1689743766733877248\n",
    "    \"California will be sending search and rescue \" \\\n",
    "    \"teams to assist in Hawaii's recovery efforts. The wildfires and \" \\\n",
    "    \"devastation that Maui is experiencing is all too familiar and all \" \\\n",
    "    \"too horrifying. We stand at the ready to aid Hawaii in its time of need.\",\n",
    "    # https://twitter.com/GavinNewsom/status/1679579172690329601\n",
    "    \"A school board in Temecula decided to reject a \" \\\n",
    "    \"textbook because it mentioned Harvey Milk. CA is stepping in. \" \\\n",
    "    \"We’re going to purchase the book for these students—the same \" \\\n",
    "    \"one that hundreds of thousands of kids are already using. \" \\\n",
    "    \"If these extremist school board members won’t do their job, \" \\\n",
    "    \"we will — and fine them for their incompetence.\",\n",
    "    # https://twitter.com/GavinNewsom/status/1650634702271942656\n",
    "    \"North Dakota GOP have decided to force women to give birth. Even victims of rape. \" \\\n",
    "    \"Meanwhile, they voted against providing school meals because child hunger isn't \" \\\n",
    "    \"\\\"the responsibility of the state.\\\"\" \\\n",
    "    \"Mandating birth is state responsibility. Helping feed those kids is not. Got it.\",\n",
    "    # https://twitter.com/GavinNewsom/status/1643745476662132737\n",
    "    \"Met with some librarians today while in Florida. \" \\\n",
    "    \"They shared with me the rich, diverse background of the \" \\\n",
    "    \"town and what's at stake if we ban our kids from learning our real history.\"\n",
    "]\n",
    "\n",
    "[model.classify(t) for t in newsom_tweets]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da68dbb7-8a24-4408-8258-4c1572b1c9cf",
   "metadata": {},
   "source": [
    "Neat! This small set classified 90% correctly. Further exploration with more robust Twitter data sets is warranted."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26d234f8-9fbd-4749-9b2d-ca23225c797a",
   "metadata": {},
   "source": [
    "## Closing Thoughts\n",
    "Even after seeing the results, I still have a hard time believing that this works so well! It feels like it _shouldn't_ work at all. But, zooming out, there are a lot of relevant factors to consider. First, there just aren't that many people writing campaign materials. It makes sense that word-choice and writing style would exhibit predictible patterns. Second, campaign emails have been A/B tested into oblivion, so there's a systematic process that cranks out similar-sounding copy. Third, the recipients of these emails have largely self-sorted. This likely bolsters the expected structure and copy uniquely for each label. Ultimately, compression algorithms optimize on patterns and predictibility. What this shows us is that the two parties are uniquely _predictible_ in their written communications.\n",
    "\n",
    "The idea of classification by compression is not new; Russell and Norvig wrote about it in 1995 in the venerable [Artificial Intelligence: A Modern Approach](https://aima.cs.berkeley.edu/3rd-ed/):\n",
    "\n",
    "![Classification by data compression](aiama.png)\n",
    "\n",
    "More recently, the [\"gzip beats BERT\" paper](https://aclanthology.org/2023.findings-acl.426/) got a lot of attention. What the **BIDEN** model demonstrates is that this technique is effective and likely generalizable on modern partisan texts.\n",
    "\n",
    "It's worth articulating again how fast and simple this method is. No GPUs. No Neural Networks. No N-grams. No transformers. No kNN.\n",
    "\n",
    "I think that's pretty cool!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a9754c3-bfc8-4c2a-979d-43e9a387ee9e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}