07_Named_Entity_Recognition_(NER).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Named Entity Recognition (NER)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "\n",
    "In this guide, you will learn about an advanced Natural Language Processing technique called Named Entity Recognition, or 'NER'.\n",
    "\n",
    "NER is an NLP task used to identify important named entities in the text such as people, places, organizations, date, or any other category. It can be used alone, or alongside topic identification, and adds a lot of semantic knowledge to the content, enabling us to understand the subject of any given text.\n",
    "\n",
    "Let us start with loading the required libraries and modules. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the Required Libraries and Modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package wordnet to\n",
      "[nltk_data]     C:\\Users\\Sadnan\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\Sadnan\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "[nltk_data] Downloading package words to\n",
      "[nltk_data]     C:\\Users\\Sadnan\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Unzipping corpora\\words.zip.\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     C:\\Users\\Sadnan\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
      "[nltk_data]       date!\n",
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\Sadnan\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package maxent_ne_chunker to\n",
      "[nltk_data]     C:\\Users\\Sadnan\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Unzipping chunkers\\maxent_ne_chunker.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "from collections import Counter\n",
    "\n",
    "nltk.download('wordnet')  #download if using this module for the first time\n",
    "\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "nltk.download('stopwords')    #download if using this module for the first time\n",
    "\n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "from nltk import word_tokenize, pos_tag, ne_chunk\n",
    "\n",
    "nltk.download('words')\n",
    "\n",
    "nltk.download('averaged_perceptron_tagger')\n",
    "\n",
    "nltk.download('punkt')\n",
    "\n",
    "nltk.download('maxent_ne_chunker')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Avengers: Endgame is a 2019 American superhero film based on the Marvel Comics superhero team the Avengers, produced by Marvel Studios and distributed by Walt Disney Studios Motion Pictures. The movie features an ensemble cast including Robert Downey Jr., Chris Evans, Mark Ruffalo, Chris Hemsworth, and others. (Source: wikipedia).\n"
     ]
    }
   ],
   "source": [
    "textexample = \"Avengers: Endgame is a 2019 American superhero film based on the Marvel Comics superhero team the Avengers, produced by Marvel Studios and distributed by Walt Disney Studios Motion Pictures. The movie features an ensemble cast including Robert Downey Jr., Chris Evans, Mark Ruffalo, Chris Hemsworth, and others. (Source: wikipedia).\"\n",
    "\n",
    "print(textexample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word Tokenization\n",
    "\n",
    "The first step is to tokenize the text into sentences which is done in the first line of code below. The second line performs word tokenization on the sentences, while the third line prints the tokenized sentence. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Avengers',\n",
       "  ':',\n",
       "  'Endgame',\n",
       "  'is',\n",
       "  'a',\n",
       "  '2019',\n",
       "  'American',\n",
       "  'superhero',\n",
       "  'film',\n",
       "  'based',\n",
       "  'on',\n",
       "  'the',\n",
       "  'Marvel',\n",
       "  'Comics',\n",
       "  'superhero',\n",
       "  'team',\n",
       "  'the',\n",
       "  'Avengers',\n",
       "  ',',\n",
       "  'produced',\n",
       "  'by',\n",
       "  'Marvel',\n",
       "  'Studios',\n",
       "  'and',\n",
       "  'distributed',\n",
       "  'by',\n",
       "  'Walt',\n",
       "  'Disney',\n",
       "  'Studios',\n",
       "  'Motion',\n",
       "  'Pictures',\n",
       "  '.'],\n",
       " ['The',\n",
       "  'movie',\n",
       "  'features',\n",
       "  'an',\n",
       "  'ensemble',\n",
       "  'cast',\n",
       "  'including',\n",
       "  'Robert',\n",
       "  'Downey',\n",
       "  'Jr.',\n",
       "  ',',\n",
       "  'Chris',\n",
       "  'Evans',\n",
       "  ',',\n",
       "  'Mark',\n",
       "  'Ruffalo',\n",
       "  ',',\n",
       "  'Chris',\n",
       "  'Hemsworth',\n",
       "  ',',\n",
       "  'and',\n",
       "  'others',\n",
       "  '.'],\n",
       " ['(', 'Source', ':', 'wikipedia', ')', '.']]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences = nltk.sent_tokenize(textexample)\n",
    "\n",
    "tokenized_sentence = [nltk.word_tokenize(sent) for sent in sentences]\n",
    "\n",
    "tokenized_sentence "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parts of Speech (POS) Tagging\n",
    "\n",
    "Parts-of-speech tagging, also called grammatical tagging, is the process of marking up a word in a text as corresponding to a particular part of speech, based on both its definition and its context. The line of code below takes the tokenized text and passes it to the 'nltk.pos_tag' function to create its POS tagging. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos_tagging_sentences = [nltk.pos_tag(sent) for sent in tokenized_sentence]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us combine these two steps into a function and analyse the output. The first to fourth lines of code below creates the function to tokenize the text and perform POS tagging. The fifth line of code runs the function to our text, while the sixth line prints the output. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Avengers', 'NNS'),\n",
       " (':', ':'),\n",
       " ('Endgame', 'NN'),\n",
       " ('is', 'VBZ'),\n",
       " ('a', 'DT'),\n",
       " ('2019', 'JJ'),\n",
       " ('American', 'JJ'),\n",
       " ('superhero', 'NN'),\n",
       " ('film', 'NN'),\n",
       " ('based', 'VBN'),\n",
       " ('on', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('Marvel', 'NNP'),\n",
       " ('Comics', 'NNP'),\n",
       " ('superhero', 'NN'),\n",
       " ('team', 'NN'),\n",
       " ('the', 'DT'),\n",
       " ('Avengers', 'NNPS'),\n",
       " (',', ','),\n",
       " ('produced', 'VBN'),\n",
       " ('by', 'IN'),\n",
       " ('Marvel', 'NNP'),\n",
       " ('Studios', 'NNP'),\n",
       " ('and', 'CC'),\n",
       " ('distributed', 'VBN'),\n",
       " ('by', 'IN'),\n",
       " ('Walt', 'NNP'),\n",
       " ('Disney', 'NNP'),\n",
       " ('Studios', 'NNP'),\n",
       " ('Motion', 'NNP'),\n",
       " ('Pictures', 'NNP'),\n",
       " ('.', '.'),\n",
       " ('The', 'DT'),\n",
       " ('movie', 'NN'),\n",
       " ('features', 'VBZ'),\n",
       " ('an', 'DT'),\n",
       " ('ensemble', 'JJ'),\n",
       " ('cast', 'NN'),\n",
       " ('including', 'VBG'),\n",
       " ('Robert', 'NNP'),\n",
       " ('Downey', 'NNP'),\n",
       " ('Jr.', 'NNP'),\n",
       " (',', ','),\n",
       " ('Chris', 'NNP'),\n",
       " ('Evans', 'NNP'),\n",
       " (',', ','),\n",
       " ('Mark', 'NNP'),\n",
       " ('Ruffalo', 'NNP'),\n",
       " (',', ','),\n",
       " ('Chris', 'NNP'),\n",
       " ('Hemsworth', 'NNP'),\n",
       " (',', ','),\n",
       " ('and', 'CC'),\n",
       " ('others', 'NNS'),\n",
       " ('.', '.'),\n",
       " ('(', '('),\n",
       " ('Source', 'NN'),\n",
       " (':', ':'),\n",
       " ('wikipedia', 'NN'),\n",
       " (')', ')'),\n",
       " ('.', '.')]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def preprocess(text):\n",
    "\n",
    "    text = nltk.word_tokenize(text)\n",
    "\n",
    "    text = nltk.pos_tag(text)\n",
    "\n",
    "    return text\n",
    "\n",
    "\n",
    "processed_text = preprocess(textexample)\n",
    "\n",
    "processed_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output above shows that every token has been tagged to its parts of speech. Some of the common abbreviations are explained below:\n",
    "\n",
    "    DT: determiner\n",
    "    IN: preposition/subordinating conjunction\n",
    "    JJ: adjective ‘big’\n",
    "    JJR: adjective, comparative ‘bigger’\n",
    "    JJS: adjective, superlative ‘biggest’\n",
    "    LS: list marker\n",
    "    NN: noun, singular ‘desk’\n",
    "    NNS: noun plural ‘desks’\n",
    "    NNP: proper noun, singular ‘Harrison’\n",
    "    NNPS: proper noun, plural ‘Americans’\n",
    "    PRP: personal pronoun I, he, she\n",
    "    RB: adverb very, silently,\n",
    "    UH: interjection\n",
    "    VB: verb, base form take\n",
    "    VBD: verb, past tense took"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chunking\n",
    "\n",
    "Once we have completed the parts-of-speech tagging, we will perform chunking. In simple terms, what chunking does is that it adds more structure to the sentence over and above the tagging. The output results in grouping of words called 'chunks'.\n",
    "\n",
    "We will perform chunking to the processed text which is done in the first line of code below. The second to fourth lines of code does the chunking, and in our example, we will only look at Nouns for the NER tagging. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Avengers/NNS\n",
      "  Endgame/NN\n",
      "  superhero/NN\n",
      "  film/NN\n",
      "  (ORGANIZATION Marvel/NNP Comics/NNP)\n",
      "  superhero/NN\n",
      "  team/NN\n",
      "  (ORGANIZATION Avengers/NNPS)\n",
      "  (PERSON Marvel/NNP Studios/NNP)\n",
      "  (PERSON Walt/NNP Disney/NNP Studios/NNP)\n",
      "  Motion/NNP\n",
      "  Pictures/NNP\n",
      "  movie/NN\n",
      "  cast/NN\n",
      "  (PERSON Robert/NNP Downey/NNP Jr./NNP)\n",
      "  (PERSON Chris/NNP Evans/NNP)\n",
      "  (PERSON Mark/NNP Ruffalo/NNP)\n",
      "  (PERSON Chris/NNP Hemsworth/NNP)\n",
      "  others/NNS\n",
      "  (PERSON Source/NN)\n",
      "  wikipedia/NN\n"
     ]
    }
   ],
   "source": [
    "res_chunk = ne_chunk(processed_text)\n",
    "\n",
    "\n",
    "for x in str(res_chunk).split('\\n'):\n",
    "\n",
    "    if '/NN' in x:\n",
    "\n",
    "        print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us explore the above output. We observe that the word tokens 'Endgame', 'film', and 'Source' are tagged as singular noun 'NN', while tokens like 'Avengers' and 'others' are tagged as plural noun 'NNS. Also, note that the names of the actors 'Robert', 'Evans', etc., have been tagged as proper noun 'NNP'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "In this guide, you have learned about how to perform Named Entity Recognition using nltk. You learned about the three important stages of Word Tokenization, POS Tagging, and Chunking that are needed to perform NER analysis. "
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "63f9911de6f9547d26d6ccd5b997d4c24c9ba8fc8843a26db797dceee8b60692"
  },
  "kernelspec": {
   "display_name": "Python 3.10.4 64-bit (windows store)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}