paddlenlp_updated_notes_English/Text Similarity/2.1 - wrapped_up_data_preprocessor.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "250b1a78",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Author: Zhengxiang (Jack) Wang \n",
    "# Date: 2022-01-15\n",
    "# GitHub: https://github.com/jaaack-wang "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4c0e463",
   "metadata": {},
   "source": [
    "## Quick start\n",
    "\n",
    "We used a lengthy tutorial to illustrate how to convert text dataset for training text matching models. However, if there are handy wrapped up functions, as available in many deep learning frameworks, this process can be easily done with a few lines of code. The following is an illustration, although in real projects, you many need additional lines of code to suit your specific needs. \n",
    "\n",
    "The following is a quick start. A more elaborated explanation is given afterwards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9bb7f78c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Two vocabulary dictionaries have been built!\n",
      "Please call \u001b[1mX.vocab_to_idx | X.idx_to_vocab\u001b[0m to find out more where [X] stands for the name you used for this TextVectorizer class.\n",
      "Number of 47 batches created!\n"
     ]
    }
   ],
   "source": [
    "from utils import *\n",
    "\n",
    "# ---- load dataset ----\n",
    "train_set = load_dataset('train.txt')\n",
    "\n",
    "# ---- numericalize the train set ----\n",
    "V = TextVectorizer(tokenize) \n",
    "text = gather_text(train_set) # for collecting texts from train set\n",
    "V.build_vocab(text) # for building mapping vocab_to_idx dictionary and text_encoder\n",
    "train_set_encoded = list(encode_dataset(train_set, encoder=V.text_encoder)) # encodoing train set\n",
    "\n",
    "# ---- build mini batches for the train set ----\n",
    "train_set_batched = build_batches(train_set_encoded, batch_size=64, include_seq_len=True)\n",
    "print(f\"Number of {len(train_set_batched)} batches created!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd3341da",
   "metadata": {},
   "source": [
    "## Intro to the wrapped functions\n",
    "\n",
    "As we will not rewrite or copy and paste (a) same functions over and over again to just load and preprocess data throughout the tutorials, some wrapped functions are therefore provided. These wrapped functions can be seen in the `utils.py` file in the same folder.\n",
    "\n",
    "These wrapped functions mostly come from the last tutorial (see: `2 - preprocess_data.ipynb`) with some revisions to make some functons more reusable. Let's do what we do in the last tutorial and this tutorial will introduce these wrapped up functions along the way."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3122b017",
   "metadata": {},
   "source": [
    "## Load dataset\n",
    "\n",
    "The wrapped up `load_dataset` allows:\n",
    "\n",
    "- loading a dataset (for tutorials in this repository) given its filepath; \n",
    "- (extended) loading multiples datasets given their filepathes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8d0753d4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3000, 1000, 1000)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from utils import load_dataset\n",
    "\n",
    "train_set, dev_set, test_set = load_dataset(['train.txt', 'dev.txt', 'test.txt'])\n",
    "\n",
    "# check. should be 3000, 1000, 1000 (recall `1 - get_data.ipynb`)\n",
    "len(train_set), len(dev_set), len(test_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dd087a6",
   "metadata": {},
   "source": [
    "## Numericalize text \n",
    "\n",
    "Recall that we need to encode the text data into something numerical so that we can train models on them. As elaborated in the last tutorial, to do so, we need to have a tokenizer and a related dictionary where we can map a token to an unique index. If we want to deocde the encoded text, we will also need to have a reversed dictionary that maps an index to a token. \n",
    "\n",
    "Re-doing all these can be tedious, so this tutorials introduces a highly wrapped up class function `TextVectorizer` to make everything easy. The `TextVectorizer` class can do the following: \n",
    "\n",
    "- building the `vocab_to_idx` and `idx_to_vocab` mapping dictionaries quickly given text and tokenizer. \n",
    "- encoding and decoding any given text(s) using the tokenizer used to build the dictionaries; \n",
    "- save and load the built `vocab_to_idx` and `idx_to_vocab` dictionaries for reuse; \n",
    "\n",
    "If you have a list of tokens ready that can be gotten from the tokenizer you pass to the `TextVectorizer` class, then you can also quickly build the dictionaries from that list of tokens. \n",
    "\n",
    "`TextVectorizer` is also callable, which means that after initializing it, you can directly encode text(s) by calling one(s) to it. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f708d68",
   "metadata": {},
   "source": [
    "### Initialization \n",
    "\n",
    "To initialize the `TextVectorizer`, you must pass a tokenizer function/method to it, which takes str as input and returns a list of tokens. You can also pass a text preprocessor function/method to it to preprocess a given text before it is being tokenized by the tokenizer. Of course, you can build a tokenizer that incorprates the preprocessor and only pass the tokenizer. They are a same thing. \n",
    "\n",
    "We will use the same tokenizer we used in the last tutorial, which is saved in the `utils.py` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a63196d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import tokenize, TextVectorizer\n",
    "\n",
    "V = TextVectorizer(tokenize)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44d6d47c",
   "metadata": {},
   "source": [
    "### Building `vocab_to_idx` and `idx_to_vocab` dictionaries\n",
    "\n",
    "To build these two dictionaries, all you need is just a (list) of text(s) and then pass it/them to the `TextVectorizer.build_vocab` function. If you have a list of tokens, you can also do so by passing the tokens to `TextVectorizer.build_vocab_from_list_tks`.\n",
    "\n",
    "If you choose to use `TextVectorizer.build_vocab`, you can also choose whether to build the vocab randomly or based on the occurences of the tokens in descending order. By defaults, it will build the vocab basde on the latter. When this is the case, you can specify how many most frequnt tokens you want to keep by specifying, for example, `top=10000`. This is not an option for the random mode because that does not make sense. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc12dbe4",
   "metadata": {},
   "source": [
    "To gather the text, we will use a simple function `gather_text` that specifically gather all the \"text_a\" and \"text_b\" from our datasets into a list. \n",
    "\n",
    "**Remember that the dictionaries should be built upon the train set or some external source, but never on the dev set or test set.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0e0229e9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6000 pieces of texts gathered from the train set.\n",
      "\n",
      "Two vocabulary dictionaries have been built!\n",
      "Please call \u001b[1mX.vocab_to_idx | X.idx_to_vocab\u001b[0m to find out more where [X] stands for the name you used for this TextVectorizer class.\n"
     ]
    }
   ],
   "source": [
    "from utils import gather_text\n",
    "\n",
    "text = gather_text(train_set)\n",
    "print(f\"{len(text)} pieces of texts gathered from the train set.\\n\")\n",
    "\n",
    "V.build_vocab(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7aebfdaa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5 random examples from the vocab_to_idx dictionary\n",
      "\n",
      "genome              6909\n",
      "accomplish          5258\n",
      "facility            7479\n",
      "wedding             1032\n",
      "build               741\n",
      "\n",
      "\n",
      "5 random examples from the idx_to_vocab dictionary\n",
      "\n",
      "3816                mobility\n",
      "6394                practicing\n",
      "5614                cfl\n",
      "4865                retrograde\n",
      "6659                tablet\n"
     ]
    }
   ],
   "source": [
    "from random import sample, seed\n",
    "# check\n",
    "\n",
    "seed(543)\n",
    "\n",
    "tmp = \"{:20}{}\"\n",
    "print(\"5 random examples from the vocab_to_idx dictionary\\n\")\n",
    "\n",
    "for item in sample(list(V.vocab_to_idx.items()), 5):\n",
    "    print(tmp.format(*item))\n",
    "    \n",
    "    \n",
    "print(\"\\n\\n5 random examples from the idx_to_vocab dictionary\\n\")\n",
    "\n",
    "for idx, tk in sample(list(V.idx_to_vocab.items()), 5):\n",
    "    print(tmp.format(str(idx), tk))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae17eb6c",
   "metadata": {},
   "source": [
    "### Saving the mapping dictionaries\n",
    "\n",
    "You can use `TextVectorizer.save_vocab_as_json` which by default will save the dictionaries, if built, as `vocab_to_idx.json` and `idx_to_vocab.json` in the current working directory. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c56568c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vocab_to_idx.json has been successfully saved!\n",
      "idx_to_vocab.json has been successfully saved!\n"
     ]
    }
   ],
   "source": [
    "V.save_vocab_as_json()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ea79823",
   "metadata": {},
   "source": [
    "### Reusing the mapping dictionaries\n",
    "\n",
    "When you need to reuse them, simply call `TextVectorizer.load_vocab_from_json`. If you do specify the filepathes to the two dictionaries, it will search for `vocab_to_idx.json` and `idx_to_vocab.json` in the current working directory and return them if any of them exists. \n",
    "\n",
    "Below, we first empty the two mapping dictionaries and reload them from the json files we saved just now. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0c30697b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vocab_to_idx.json has been successfully loaded! Please call \u001b[1mX.vocab_to_idx\u001b[0m to find out more.\n",
      "idx_to_vocab.json has been successfully loaded! Please call \u001b[1mX.idx_to_vocab\u001b[0m to find out more.\n",
      "\n",
      "Where [X] stands for the name you used for this TextVectorizer class.\n"
     ]
    }
   ],
   "source": [
    "V.vocab_to_idx, V.idx_to_vocab = None, None\n",
    "\n",
    "V.load_vocab_from_json()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50f23dd4",
   "metadata": {},
   "source": [
    "### Encoding and decoding text\n",
    "\n",
    "- Encoding: call initialized `TextVectorizer` or `TextVectorizer.text_encoder`.\n",
    "- Decoding: call `TextVectorizer.text_decoder`.\n",
    "\n",
    "Let's use some sample texts from the dev set as examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "fdce19d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original: How do I stop my Pit Bull/English Bulldog mix from biting my shoes?\n",
      "Encoded: [5, 10, 6, 172, 19, 1, 1, 1, 607, 33, 1, 19, 1981]\n",
      "Decoded: how do i stop my [UNK] [UNK] [UNK] mix from [UNK] my shoes\n",
      "\n",
      "Original: How much does it cost to patent something?\n",
      "Encoded: [5, 102, 21, 20, 323, 8, 1971, 326]\n",
      "Decoded: how much does it cost to patent something\n",
      "\n",
      "Original: Why am I not comfortable with the caste-based reservation system in India?\n",
      "Encoded: [16, 77, 6, 57, 6218, 31, 2, 7515, 602, 194, 9, 37]\n",
      "Decoded: why am i not comfortable with the castebased reservation system in india\n",
      "\n",
      "['how do i stop my [UNK] [UNK] [UNK] mix from [UNK] my shoes', 'how much does it cost to patent something', 'why am i not comfortable with the castebased reservation system in india']\n"
     ]
    }
   ],
   "source": [
    "dev_text = gather_text(dev_set)\n",
    "sample_texts = sample(dev_text, 3)\n",
    "\n",
    "assert len(dev_text) == len(dev_set) * 2 # text_a plus text_b\n",
    "\n",
    "seed(834)\n",
    "\n",
    "for text in sample_texts:\n",
    "    encoded = V.text_encoder(text) # or V(text)\n",
    "    decoded = V.text_decoder(encoded)\n",
    "    print(\"Original:\", text)\n",
    "    print(\"Encoded:\", encoded)\n",
    "    print(\"Decoded:\", decoded)\n",
    "    print()\n",
    "    \n",
    "# Or pass a list of text or text_ids to encode or decode\n",
    "print(V.text_decoder(V(sample_texts)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62fb0760",
   "metadata": {},
   "source": [
    "### Encode the train set\n",
    "\n",
    "We will reuse the `encode_dataset` built in the last tutorial to encode the train set (or dev set for validation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "76e3868d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[5, 10, 6, 223, 7, 43, 852], [5, 10, 6, 223, 34, 852, 9, 94], 1]\n"
     ]
    }
   ],
   "source": [
    "from utils import encode_dataset\n",
    "\n",
    "train_set_encoded = list(encode_dataset(train_set, encoder=V.text_encoder))\n",
    "\n",
    "# check\n",
    "print(train_set_encoded[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6ea2774",
   "metadata": {},
   "source": [
    "## Building batches\n",
    "\n",
    "We will reuse the `build_batches` built in the last tutorial to build the mini batches for the train set (or dev set for validation). The parameters are same except one `include_seq_len` that is added specifically for "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "40ae7d98",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of 47 batches created!\n"
     ]
    }
   ],
   "source": [
    "from utils import build_batches\n",
    "\n",
    "train_set_batched = build_batches(train_set_encoded, batch_size=64, include_seq_len=True)\n",
    "\n",
    "print(f\"Number of {len(train_set_batched)} batches created!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c51c4e44",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------- # 1 batch ----------\n",
      "Shape of text_a batch: (64, 23)\n",
      "Shape of text_b batch: (64, 45)\n",
      "Shape of text_a_len batch: (64,)\n",
      "Shape of text_b_len batch: (64,)\n",
      "Shape of label batch: (64,)\n",
      "\n",
      "---------- # 2 batch ----------\n",
      "Shape of text_a batch: (64, 30)\n",
      "Shape of text_b batch: (64, 29)\n",
      "Shape of text_a_len batch: (64,)\n",
      "Shape of text_b_len batch: (64,)\n",
      "Shape of label batch: (64,)\n",
      "\n",
      "---------- # 3 batch ----------\n",
      "Shape of text_a batch: (64, 28)\n",
      "Shape of text_b batch: (64, 36)\n",
      "Shape of text_a_len batch: (64,)\n",
      "Shape of text_b_len batch: (64,)\n",
      "Shape of label batch: (64,)\n",
      "\n",
      "---------- # 4 batch ----------\n",
      "Shape of text_a batch: (64, 26)\n",
      "Shape of text_b batch: (64, 27)\n",
      "Shape of text_a_len batch: (64,)\n",
      "Shape of text_b_len batch: (64,)\n",
      "Shape of label batch: (64,)\n",
      "\n",
      "---------- # 5 batch ----------\n",
      "Shape of text_a batch: (64, 21)\n",
      "Shape of text_b batch: (64, 35)\n",
      "Shape of text_a_len batch: (64,)\n",
      "Shape of text_b_len batch: (64,)\n",
      "Shape of label batch: (64,)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# check\n",
    "\n",
    "for idx, batch in enumerate(train_set_batched[:5]):\n",
    "    a, b, a_l, b_l, l = batch\n",
    "    print(f\"{'-' * 10} # {idx+1} batch {'-' * 10}\")\n",
    "    \n",
    "    print(\"Shape of text_a batch:\", a.shape)\n",
    "    print(\"Shape of text_b batch:\", b.shape)\n",
    "    print(\"Shape of text_a_len batch:\", a_l.shape)\n",
    "    print(\"Shape of text_b_len batch:\", b_l.shape)\n",
    "    print(\"Shape of label batch:\", l.shape)\n",
    "    print()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}