tokenization.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenize text\n",
    "\n",
    "This sample shows how to use the [Microsoft.ML.Tokenizers](https://www.nuget.org/packages/Microsoft.ML.Tokenizers/) library to tokenize text and get information about token counts. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Microsoft.ML.Tokenizers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>Microsoft.ML.Tokenizers, 0.21.1</span></li></ul></div></div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#r \"nuget:Microsoft.ML.Tokenizers\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add using statements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "using Microsoft.ML.Tokenizers;"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download and define vocab resources\n",
    "\n",
    "Download the following files and place them in the root directory. These vocabulary files are what are used to encode the text into tokens. \n",
    "\n",
    "- [GPT Vocabulary Files](https://huggingface.co/gpt2/tree/main)\n",
    "  - vocab.json\n",
    "  - merges.txt "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "var vocabFilePath = @\"vocab.json\";\n",
    "var mergeFilePath = @\"merges.txt\";"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize Tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "var tokenizer = new Tokenizer(new Bpe(vocabFilePath, mergeFilePath));"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encode text into tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<details open=\"open\" class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>Microsoft.ML.Tokenizers.TokenizerResult</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>OriginalString</td><td><div class=\"dni-plaintext\"><pre>the brown fox jumped over the lazy dog!</pre></div></td></tr><tr><td>NormalizedString</td><td><div class=\"dni-plaintext\"><pre>the brown fox jumped over the lazy dog!</pre></div></td></tr><tr><td>OffsetsMappedToOriginalString</td><td><div class=\"dni-plaintext\"><pre>True</pre></div></td></tr><tr><td>Ids</td><td><div class=\"dni-plaintext\"><pre>[ 1169, 33282, 12792, 73, 27073, 2502, 1169, 75, 12582, 9703, 0 ]</pre></div></td></tr><tr><td>Tokens</td><td><div class=\"dni-plaintext\"><pre>[ the, brown, fox, j, umped, over, the, l, azy, dog, ! ]</pre></div></td></tr><tr><td>Offsets</td><td><table><thead><tr><th><i>index</i></th><th>value</th></tr></thead><tbody><tr><td>0</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>(0, 3)</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>Item1</td><td><div class=\"dni-plaintext\"><pre>0</pre></div></td></tr><tr><td>Item2</td><td><div class=\"dni-plaintext\"><pre>3</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>1</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>(4, 9)</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>Item1</td><td><div class=\"dni-plaintext\"><pre>4</pre></div></td></tr><tr><td>Item2</td><td><div class=\"dni-plaintext\"><pre>9</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>2</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>(10, 13)</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>Item1</td><td><div class=\"dni-plaintext\"><pre>10</pre></div></td></tr><tr><td>Item2</td><td><div class=\"dni-plaintext\"><pre>13</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>3</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>(14, 15)</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>Item1</td><td><div class=\"dni-plaintext\"><pre>14</pre></div></td></tr><tr><td>Item2</td><td><div class=\"dni-plaintext\"><pre>15</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>4</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>(15, 20)</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>Item1</td><td><div class=\"dni-plaintext\"><pre>15</pre></div></td></tr><tr><td>Item2</td><td><div class=\"dni-plaintext\"><pre>20</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>5</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>(21, 25)</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>Item1</td><td><div class=\"dni-plaintext\"><pre>21</pre></div></td></tr><tr><td>Item2</td><td><div class=\"dni-plaintext\"><pre>25</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>6</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>(26, 29)</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>Item1</td><td><div class=\"dni-plaintext\"><pre>26</pre></div></td></tr><tr><td>Item2</td><td><div class=\"dni-plaintext\"><pre>29</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>7</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>(30, 31)</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>Item1</td><td><div class=\"dni-plaintext\"><pre>30</pre></div></td></tr><tr><td>Item2</td><td><div class=\"dni-plaintext\"><pre>31</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>8</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>(31, 34)</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>Item1</td><td><div class=\"dni-plaintext\"><pre>31</pre></div></td></tr><tr><td>Item2</td><td><div class=\"dni-plaintext\"><pre>34</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>9</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>(35, 38)</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>Item1</td><td><div class=\"dni-plaintext\"><pre>35</pre></div></td></tr><tr><td>Item2</td><td><div class=\"dni-plaintext\"><pre>38</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>10</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>(38, 39)</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>Item1</td><td><div class=\"dni-plaintext\"><pre>38</pre></div></td></tr><tr><td>Item2</td><td><div class=\"dni-plaintext\"><pre>39</pre></div></td></tr></tbody></table></div></details></td></tr></tbody></table></td></tr></tbody></table></div></details><style>\r\n",
       ".dni-code-hint {\r\n",
       "    font-style: italic;\r\n",
       "    overflow: hidden;\r\n",
       "    white-space: nowrap;\r\n",
       "}\r\n",
       ".dni-treeview {\r\n",
       "    white-space: nowrap;\r\n",
       "}\r\n",
       ".dni-treeview td {\r\n",
       "    vertical-align: top;\r\n",
       "    text-align: start;\r\n",
       "}\r\n",
       "details.dni-treeview {\r\n",
       "    padding-left: 1em;\r\n",
       "}\r\n",
       "table td {\r\n",
       "    text-align: start;\r\n",
       "}\r\n",
       "table tr { \r\n",
       "    vertical-align: top; \r\n",
       "    margin: 0em 0px;\r\n",
       "}\r\n",
       "table tr td pre \r\n",
       "{ \r\n",
       "    vertical-align: top !important; \r\n",
       "    margin: 0em 0px !important;\r\n",
       "} \r\n",
       "table th {\r\n",
       "    text-align: start;\r\n",
       "}\r\n",
       "</style>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "var input = \"the brown fox jumped over the lazy dog!\";\n",
    "var tokenizerEncodedResult = tokenizer.Encode(input);\n",
    "tokenizerEncodedResult"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get token count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"dni-plaintext\"><pre>11</pre></div><style>\r\n",
       ".dni-code-hint {\r\n",
       "    font-style: italic;\r\n",
       "    overflow: hidden;\r\n",
       "    white-space: nowrap;\r\n",
       "}\r\n",
       ".dni-treeview {\r\n",
       "    white-space: nowrap;\r\n",
       "}\r\n",
       ".dni-treeview td {\r\n",
       "    vertical-align: top;\r\n",
       "    text-align: start;\r\n",
       "}\r\n",
       "details.dni-treeview {\r\n",
       "    padding-left: 1em;\r\n",
       "}\r\n",
       "table td {\r\n",
       "    text-align: start;\r\n",
       "}\r\n",
       "table tr { \r\n",
       "    vertical-align: top; \r\n",
       "    margin: 0em 0px;\r\n",
       "}\r\n",
       "table tr td pre \r\n",
       "{ \r\n",
       "    vertical-align: top !important; \r\n",
       "    margin: 0em 0px !important;\r\n",
       "} \r\n",
       "table th {\r\n",
       "    text-align: start;\r\n",
       "}\r\n",
       "</style>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tokenizerEncodedResult.Tokens.Count()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".NET (C#)",
   "language": "C#",
   "name": ".net-csharp"
  },
  "language_info": {
   "name": "polyglot-notebook"
  },
  "polyglot_notebook": {
   "kernelInfo": {
    "defaultKernelName": "csharp",
    "items": [
     {
      "aliases": [],
      "languageName": "csharp",
      "name": "csharp"
     }
    ]
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}