BoW_TF-IDF_PyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BoW and TF-IDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchtext\n",
    "import os\n",
    "import collections\n",
    "\n",
    "from torchtext.data.utils import ngrams_iterator\n",
    "\n",
    "from torch.utils.data import DataLoader\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "\n",
    "We will start with a simple text classification task based on AG_NEWS sample dataset, which is to classify news headlines into one of 4 categories: World, Sports, Business and Sci/Tech"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root=\"./data\")\n",
    "classes =  ['World', 'Sports', 'Business', 'Sci/Tech']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.utils.data.datapipes.iter.sharding.ShardingFilterIterDataPipe"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(train_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**Sci/Tech** -> Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\band of ultra-cynics, are seeing green again.\n",
      "\n",
      "**Sci/Tech** -> Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\\which has a reputation for making well-timed and occasionally\\controversial plays in the defense industry, has quietly placed\\its bets on another part of the market.\n",
      "\n",
      "**Sci/Tech** -> Oil and Economy Cloud Stocks' Outlook (Reuters) Reuters - Soaring crude prices plus worries\\about the economy and the outlook for earnings are expected to\\hang over the stock market next week during the depth of the\\summer doldrums.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i,x in zip(range(3),train_dataset):\n",
    "    print(f\"**{classes[x[0]]}** -> {x[1]}\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
    "train_dataset = list(train_dataset)\n",
    "test_dataset = list(test_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3,\n",
       " \"Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\\\band of ultra-cynics, are seeing green again.\")"
      ]
     },
     "execution_count": 166,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dataset[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove \\\\\n",
    "dumi_train = []\n",
    "dumi_test = []\n",
    "for i in range(len(train_dataset)):\n",
    "    dumi_train.append(list(train_dataset[i]))\n",
    "for i in range(len(test_dataset)):\n",
    "    dumi_test.append(list(test_dataset[i]))\n",
    "\n",
    "for i in range(len(dumi_train)):\n",
    "    dumi_train[i][1] = dumi_train[i][1].replace(\"\\\\\",\" \")\n",
    "for i in range(len(dumi_test)):\n",
    "    dumi_test[i][1] = dumi_test[i][1].replace(\"\\\\\",\" \")\n",
    "\n",
    "for i in range(len(dumi_train)):\n",
    "    dumi_train[i] = tuple(dumi_train[i])\n",
    "for i in range(len(dumi_test)):\n",
    "    dumi_test[i] = tuple(dumi_test[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset = dumi_train\n",
    "test_dataset = dumi_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization and vectorization\n",
    "\n",
    "* <code>torchtext.data.utils.get_tokenizer</code>\n",
    "* <code>torchtext.vocab,vocab</code>\n",
    "* <code>collection.Count</code>\n",
    "* encode and decode with <code>vocab.get_stoi()[string]</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = torchtext.data.utils.get_tokenizer(\"basic_english\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "first token list:\n",
      "['wall', 'st', '.', 'bears', 'claw', 'back', 'into', 'the', 'black', '(', 'reuters', ')', 'reuters', '-', 'short-sellers', ',', 'wall', 'street', \"'\", 's', 'dwindling', 'band', 'of', 'ultra-cynics', ',', 'are', 'seeing', 'green', 'again', '.']\n"
     ]
    }
   ],
   "source": [
    "first_sentence = train_dataset[0][1]\n",
    "f_tokens = tokenizer(first_sentence)\n",
    "print(f'\\nfirst token list:\\n{f_tokens}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "len of dicctionary :  87235\n"
     ]
    }
   ],
   "source": [
    "counter = collections.Counter()\n",
    "for (label, line) in train_dataset:\n",
    "    counter.update(tokenizer(line))\n",
    "vocab = torchtext.vocab.vocab(counter, min_freq=1)\n",
    "print(\"len of dicctionary : \", len(vocab))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Index lockup in 1st sentence:\n",
      "[[0, 'wall'], [1, 'st'], [2, '.'], [3, 'bears'], [4, 'claw'], [5, 'back'], [6, 'into'], [7, 'the'], [8, 'black'], [9, '('], [10, 'reuters'], [11, ')'], [10, 'reuters'], [12, '-'], [13, 'short-sellers'], [14, ','], [0, 'wall'], [15, 'street'], [16, \"'\"], [17, 's'], [18, 'dwindling'], [19, 'band'], [20, 'of'], [21, 'ultra-cynics'], [14, ','], [22, 'are'], [23, 'seeing'], [24, 'green'], [25, 'again'], [2, '.']]\n"
     ]
    }
   ],
   "source": [
    "word_lookup = [list((vocab[w], w)) for w in f_tokens]\n",
    "print(f'\\nIndex lockup in 1st sentence:\\n{word_lookup}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 10, 12, 13, 14, 0, 15, 16, 17, 18, 19, 20, 21, 14, 22, 23, 24, 25, 2]\n"
     ]
    }
   ],
   "source": [
    "def encode(x):\n",
    "    return [vocab.get_stoi()[s] for s in tokenizer(x)]\n",
    "\n",
    "vec = encode(first_sentence)\n",
    "print(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['wall',\n",
       " 'st',\n",
       " '.',\n",
       " 'bears',\n",
       " 'claw',\n",
       " 'back',\n",
       " 'into',\n",
       " 'the',\n",
       " 'black',\n",
       " '(',\n",
       " 'reuters',\n",
       " ')',\n",
       " 'reuters',\n",
       " '-',\n",
       " 'short-sellers',\n",
       " ',',\n",
       " 'wall',\n",
       " 'street',\n",
       " \"'\",\n",
       " 's',\n",
       " 'dwindling',\n",
       " 'band',\n",
       " 'of',\n",
       " 'ultra-cynics',\n",
       " ',',\n",
       " 'are',\n",
       " 'seeing',\n",
       " 'green',\n",
       " 'again',\n",
       " '.']"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def decode(x):\n",
    "    return [vocab.get_itos()[i] for i in x]\n",
    "\n",
    "decode(vec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BiGrams, TriGrams and N-Grams\n",
    "\n",
    "One limitation of word tokenization is that some words are part of multi word expressions, for example, the word _'hot dog'_ has a completely different meaning than the words 'hot' and 'dog' in other contexts. If we represent words 'hot` and 'dog' always by the same vectors, it can confuse the model.\n",
    "\n",
    "To address this, **N-gram representations** are sometimes used in document classification, where the frequency of each word, bi-word or tri-word is a useful feature for training classifiers. \n",
    "\n",
    "To get n-gram representation, we can use `ngrams_iterator` function that will convert the sequence of tokens to the sequence of n-grams. In practice, n-gram vocabulary size is still too high to represent words as one-hot vectors, and thus we need to combine this representation with some dimensionality reduction techniques, such as **embeddings**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bigram vocab size = 505241\n"
     ]
    }
   ],
   "source": [
    "bi_counter = collections.Counter()\n",
    "for (label, line) in train_dataset:\n",
    "    bi_counter.update(ngrams_iterator(tokenizer(line), \n",
    "                                      ngrams=2))\n",
    "\n",
    "bi_vocab = torchtext.vocab.vocab(bi_counter, \n",
    "                                 min_freq=2)\n",
    "print(f\"Bigram vocab size = {len(bi_vocab)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0,\n",
       " 1,\n",
       " 2,\n",
       " 3,\n",
       " 4,\n",
       " 5,\n",
       " 6,\n",
       " 7,\n",
       " 8,\n",
       " 9,\n",
       " 10,\n",
       " 11,\n",
       " 10,\n",
       " 12,\n",
       " 13,\n",
       " 14,\n",
       " 0,\n",
       " 15,\n",
       " 16,\n",
       " 17,\n",
       " 18,\n",
       " 19,\n",
       " 20,\n",
       " 21,\n",
       " 14,\n",
       " 22,\n",
       " 23,\n",
       " 24,\n",
       " 25,\n",
       " 2]"
      ]
     },
     "execution_count": 206,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def encode(x):\n",
    "    return [bi_vocab.get_stoi()[s] for s in tokenizer(x) if s in bi_vocab ]\n",
    "\n",
    "encode(first_sentence)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bag of Words and TF-IDF\n",
    "\n",
    "Bag of Words (BoW) vector representation is the most commonly used traditional vector representation. Each word is linked to a vector index, vector element contains the number of occurrences of a word in a given document.\n",
    "> You can also think of BoW as a sum of all one-hot-encoded vectors for individual words in the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sample text:\n",
      "Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling band of ultra-cynics, are seeing green again.\n",
      "\n",
      "BoW vector:\n",
      "tensor([2., 1., 2.,  ..., 0., 0., 0.])\n"
     ]
    }
   ],
   "source": [
    "vocab_size = 10000\n",
    "def to_bow(text,bow_vocab_size=vocab_size):#len(bi_vocab) 505241\n",
    "    res = torch.zeros(bow_vocab_size,dtype=torch.float32)\n",
    "    for i in encode(text):\n",
    "        if i<bow_vocab_size:\n",
    "            res[i] += 1\n",
    "    return res\n",
    "\n",
    "print(f\"sample text:\\n{train_dataset[0][1]}\")\n",
    "print(f\"\\nBoW vector:\\n{to_bow(train_dataset[0][1])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training BoW classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bowify(b):\n",
    "    return (\n",
    "            torch.LongTensor([t[0]-1 for t in b]),\n",
    "            torch.stack([to_bow(t[1]) for t in b])\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3, \"Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling band of ultra-cynics, are seeing green again.\")\n",
      "(3, 'Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group, which has a reputation for making well-timed and occasionally controversial plays in the defense industry, has quietly placed its bets on another part of the market.')\n"
     ]
    }
   ],
   "source": [
    "print(train_dataset[0])\n",
    "print(train_dataset[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([2, 2]),\n",
       " tensor([[2., 1., 2.,  ..., 0., 0., 0.],\n",
       "         [0., 0., 1.,  ..., 0., 0., 0.]]))"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bowify([train_dataset[0],train_dataset[1]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_loader = DataLoader(train_dataset, batch_size=16, collate_fn=bowify, shuffle=True)\n",
    "test_loader = DataLoader(test_dataset, batch_size=16, collate_fn=bowify, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [],
   "source": [
    "net = torch.nn.Sequential(torch.nn.Linear(vocab_size,4),\n",
    "                          torch.nn.LogSoftmax(dim=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.NLLLoss(),epoch_size=None, report_freq=200):\n",
    "    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)\n",
    "    net.train()\n",
    "    total_loss,acc,count,i = 0,0,0,0\n",
    "    for labels,features in dataloader:\n",
    "        optimizer.zero_grad()\n",
    "        out = net(features)\n",
    "        loss = loss_fn(out,labels) #cross_entropy(out,labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        total_loss+=loss\n",
    "        _,predicted = torch.max(out,1)\n",
    "        acc+=(predicted==labels).sum()\n",
    "        count+=len(labels)\n",
    "        i+=1\n",
    "        if i%report_freq==0:\n",
    "            print(f\"{count}: acc={acc.item()/count}\")\n",
    "        if epoch_size and count>epoch_size:\n",
    "            break\n",
    "    return total_loss.item()/count, acc.item()/count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.08743439614772797, 0.3125)"
      ]
     },
     "execution_count": 208,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_epoch(net,train_loader,epoch_size=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Term Frequency / Inverse Document Frequency  -  (TF-IDF)\n",
    "\n",
    "In BoW representation, word occurrences are evenly weighted, regardless of the word itself. However, it is clear that frequent words, such as *'a'*, *'in'*, *'the'* etc. are much less important for the classification, than specialized terms. In fact, in most NLP tasks some words are more relevant than others.\n",
    "\n",
    "**TF-IDF** stands for **term frequency–inverse document frequency**. It is a variation of bag of words, where instead of a binary 0/1 value indicating the appearance of a word in a document, a floating-point value is used, which is related to the **_frequency of word occurrence_** in the corpus.\n",
    "\n",
    "The formula to calculate TF-IDF is:  $w_{ij} = tf_{ij}\\times\\log({N\\over df_i})$\n",
    "\n",
    "Here's the meaning of each parameter in the formula:\n",
    "* $i$ is the word \n",
    "* $j$ is the document\n",
    "* $w_{ij}$ is the weight or the importance of the word in the document\n",
    "* $tf_{ij}$ is the number of occurrences of the word $i$ in the document $j$, i.e. the BoW value we have seen before\n",
    "* $N$ is the number of documents in the collection\n",
    "* $df_i$ is the number of documents containing the word $i$ in the whole collection\n",
    "\n",
    "TF-IDF value $w_{ij}$ increases proportionally to the number of times a word appears in a document and is offset by the number of documents in the corpus that contains the word, which helps to adjust for the fact that some words appear more frequently than others. For example, if the word appears in *every* document in the collection, $df_i=N$, and $w_{ij}=0$, and those terms would be completely disregarded.\n",
    "\n",
    "First, let's compute document frequency $df_i$ for each word $i$. We can represent it as tensor of size `vocab_size`. We will limit the number of documents to $N=1000$ to speed up processing. For each input sentence, we compute the set of words (represented by their numbers), and increase the corresponding counter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [],
   "source": [
    "N = 100\n",
    "df = torch.zeros(vocab_size)\n",
    "for _,line in train_dataset[:N]:\n",
    "    for i in set(encode(line)):\n",
    "        df[i] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([7.0330, 3.5165, 0.1015,  ..., 0.0000, 0.0000, 0.0000])\n"
     ]
    }
   ],
   "source": [
    "def tf_idf(s):\n",
    "    bow = to_bow(s)\n",
    "    return bow*torch.log((N+1)/(df+1))\n",
    "\n",
    "print(tf_idf(train_dataset[0][1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> You may have noticed that we used a slightly different formula for TF-IDF, namely $\\log({N+1\\over df_i+1})$ instead of $\\log({N\\over df_i})$. This yields similar results, but prevents division by 0 in those cases when $df_i=0$.\n",
    "\n",
    "Even though TF-IDF representation calculates different weights to different words according to their importance, it is unable to correctly capture the meaning, largely because the order of words in the sentence is still not taken into account. As the famous linguist J. R. Firth said in 1935, “The complete meaning of a word is always contextual, and no study of meaning apart from context can be taken seriously”. We will learn in the later units how to capture contextual information from text using language modeling."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}