2.5 Tokenizing Text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.5 Tokenizing Text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fundamental step in NLP involves converting our text into smaller units through a process known as tokenization. These smaller units are known as our tokens. Word tokenization is the most common form of tokenization, where individual words in the text becomes a token, but tokens can also be sentences, sub words or individual characters depending on your use case. \n",
    "\n",
    "Why do we do this? The meaning of the overall text is better understood if we can analyse and understand the individual parts as well as the whole. It's also an important step before we vecotrize our data, which we'll cover more in the next section of this course. \n",
    "\n",
    "Now let's look at some examples of sentence and word tokenization using the nltk package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /home/lauren/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('punkt')\n",
    "from nltk.tokenize import word_tokenize, sent_tokenize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentance tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"Her cat's name is Luna.\", \"Her dog's name is max\"]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences = \"Her cat's name is Luna. Her dog's name is max\"\n",
    "sent_tokenize(sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Her', 'cat', \"'s\", 'name', 'is', 'Luna']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence = \"Her cat's name is Luna\"\n",
    "word_tokenize(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how \"cat's\" has been split into 2 tokens. This may be fine for your task but it is definitely something to keep in mind when you are preprocessing any text data - you might want to remove punctuation or replace contractions before tokenizing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Her',\n",
       " 'cat',\n",
       " \"'s\",\n",
       " 'name',\n",
       " 'is',\n",
       " 'Luna',\n",
       " 'and',\n",
       " 'her',\n",
       " 'dog',\n",
       " \"'s\",\n",
       " 'name',\n",
       " 'is',\n",
       " 'max']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_2 = \"Her cat's name is Luna and her dog's name is max\"\n",
    "word_tokenize(sentence_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These tokens illustrate what we learned in our last lesson about the importance of using lowercase. We can see we have two instances of the word 'her' - one which is capitalised. The tokens then are different and will be treated as different in most analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}