notes/programming/06_classifier.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "title: Training and evaluating classifiers\n",
    "date: 2024-03-21\n",
    "categories:\n",
    "    - python\n",
    "filters:\n",
    "    - codeblocklabel\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some support functions in the [`read_data.py` module](read_data.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from read_data import read_baby_names, \\\n",
    "    test_dev_train_split\n",
    "import nltk\n",
    "from nltk import NaiveBayesClassifier\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "baby_names = read_baby_names(\"data/baby_2017.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_name_features(\n",
    "        data_line: list[str, str]\n",
    "    ) -> list[tuple[dict, str]]:\n",
    "    gender = data_line[0]\n",
    "    name = data_line[1]\n",
    "\n",
    "    features = {\n",
    "        \"first_letter\": name[0],\n",
    "        \"second_letter\": name[1],\n",
    "        \"last_letter\": name[-1]\n",
    "    }\n",
    "\n",
    "    return (features, gender)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_name_features = [\n",
    "    make_name_features(line) \n",
    "    for line in baby_names\n",
    "    ]\n",
    "train_name, dev_name, test_name = test_dev_train_split(all_name_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we train a whole Naive Bayes classifier, let's think about how we might do with a *much* simpler classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def joes_v_good_classifier(features: dict) -> str:\n",
    "    return \"F\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's score its accuracy on the dev set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def classifier_metric(classifier, data):\n",
    "    guess = classifier(data[0])\n",
    "    answer = data[1]\n",
    "\n",
    "    if guess == answer:\n",
    "        return 1\n",
    "    return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "joe_guesses = np.array([\n",
    "    classifier_metric(joes_v_good_classifier, data)\n",
    "    for data in dev_name\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.566676932553126"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joe_guesses.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By just always guessing `\"F\"`, I did better than chance!\n",
    "\n",
    "## Moving beyond accuracy\n",
    "\n",
    "- Recall: Of all of the names that *were* `\"F\"`, how many did the classifier label `\"F\"`?\n",
    "- Precision: Of all of the names labelled `\"F\"`, how many *were* `\"F\"`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "joe_recall = np.array([\n",
    "    classifier_metric(joes_v_good_classifier, data)\n",
    "    for data in dev_name\n",
    "    if data[1] == \"F\"\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joe_recall_est = joe_recall.mean()\n",
    "joe_recall_est"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "joe_precision = np.array([\n",
    "    classifier_metric(joes_v_good_classifier, data)\n",
    "    for data in dev_name\n",
    "    if joes_v_good_classifier(data[0]) == \"F\"\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.566676932553126"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joe_precision_est = joe_precision.mean()\n",
    "joe_precision_est"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These two measure are often combined into a single score called the \"F Measure\". \n",
    "\n",
    "$$\n",
    "F = 2\\frac{pr}{p+r}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "joe_f = 2 * ((joe_recall_est * joe_precision_est)/(joe_recall_est + joe_precision_est))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7234126204049538"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joe_f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A high-precision, low-recall classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def a_classifier(features: dict) -> str:\n",
    "    if features[\"last_letter\"] == \"a\":\n",
    "        return \"F\"\n",
    "    \n",
    "    return \"M\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "a_recall = np.array([\n",
    "    classifier_metric(a_classifier, data)\n",
    "    for data in dev_name\n",
    "    if data[1] == \"F\"\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3391304347826087"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a_recall_est = a_recall.mean()\n",
    "a_recall_est"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "a_precision = np.array([\n",
    "    classifier_metric(a_classifier, data)\n",
    "    for data in dev_name\n",
    "    if a_classifier(data[0]) == \"F\"\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9292628443782577"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a_precision_est = a_precision.mean()\n",
    "a_precision_est"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The way this classifier worked, it was really reluctant to label a name `\"F\"`, but when it *did*, it was mostly right. For this data set, this results in a way worse F measure than just labelling every single name `\"F\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.49691419470435993"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a_f = 2 * ((a_recall_est * a_precision_est)/(a_recall_est + a_precision_est))\n",
    "a_f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training and evaluating the Naive Bayes classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "nb_classifier = NaiveBayesClassifier.train(train_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "nb_recall = np.array([\n",
    "    classifier_metric(nb_classifier.classify, data)\n",
    "    for data in dev_name\n",
    "    if data[1] == \"F\"\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7902173913043479"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb_recall_est = nb_recall.mean()\n",
    "nb_recall_est"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "nb_precision = np.array([\n",
    "    classifier_metric(nb_classifier.classify, data)\n",
    "    for data in dev_name\n",
    "    if nb_classifier.classify(data[0]) == \"F\"\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7859459459459459"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb_precision_est = nb_precision.mean()\n",
    "nb_precision_est"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7880758807588076"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb_f = 2 * ((nb_precision_est * nb_recall_est)/(nb_precision_est + nb_recall_est))\n",
    "nb_f"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}