codes/7_bow_tfidf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bag of Words and TF-IDF\n",
    "Below, we'll look at three useful methods of vectorizing text.\n",
    "- `CountVectorizer` - Bag of Words\n",
    "- `TfidfTransformer` - TF-IDF values\n",
    "- `TfidfVectorizer` - Bag of Words AND TF-IDF values\n",
    "\n",
    "Let's first use an example from earlier and apply the text processing steps we saw in this lesson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\ziaeeamir\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\ziaeeamir\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "[nltk_data] Downloading package wordnet to\n",
      "[nltk_data]     C:\\Users\\ziaeeamir\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "\n",
    "nltk.download('punkt')\n",
    "nltk.download('stopwords')\n",
    "nltk.download('wordnet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = [\"The first time you see The Second Renaissance it may look boring.\",\n",
    "        \"Look at it at least twice and definitely watch part 2.\",\n",
    "        \"It will change your view of the matrix.\",\n",
    "        \"Are the human people the ones who started the war?\",\n",
    "        \"Is AI a bad thing ?\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_words = stopwords.words('english')\n",
    "lemmatizer = WordNetLemmatizer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Th function `tokenize`  takes in a string of text and applies the following:\n",
    "\n",
    "- case normalization (convert to all lowercase)\n",
    "- punctuation removal\n",
    "- tokenization, lemmatization, and stop word removal using `nltk`\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize(text):\n",
    "    # normalize case and remove punctuation\n",
    "    text = re.sub(r\"[^a-zA-Z0-9]\", \" \", text.lower())\n",
    "    \n",
    "    # tokenize text\n",
    "    tokens = word_tokenize(text)\n",
    "    \n",
    "    # lemmatize and remove stop words\n",
    "    tokens = [lemmatizer.lemmatize(word) for word in tokens if word not in stop_words]\n",
    "\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `CountVectorizer` (Bag of Words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# initialize count vectorizer object\n",
    "vect = CountVectorizer(tokenizer = tokenize)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get counts of each token (word) in text data\n",
    "X = vect.fit_transform(corpus)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 0 0 1 0 0 1 0 0 1 0 1 0 0 0 1 1 1 0 0 1 0 0 0 0]\n",
      " [1 0 0 0 0 1 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 1]\n",
      " [0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0]\n",
      " [0 0 0 0 0 0 0 1 0 0 0 0 1 0 1 0 0 0 1 0 0 0 0 1 0]\n",
      " [0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0]]\n",
      "(5, 25)\n"
     ]
    }
   ],
   "source": [
    "# convert sparse matrix to numpy array to view\n",
    "print(X.toarray())\n",
    "print(X.toarray().shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'first': 6,\n",
       " 'time': 20,\n",
       " 'see': 17,\n",
       " 'second': 16,\n",
       " 'renaissance': 15,\n",
       " 'may': 11,\n",
       " 'look': 9,\n",
       " 'boring': 3,\n",
       " 'least': 8,\n",
       " 'twice': 21,\n",
       " 'definitely': 5,\n",
       " 'watch': 24,\n",
       " 'part': 13,\n",
       " '2': 0,\n",
       " 'change': 4,\n",
       " 'view': 22,\n",
       " 'matrix': 10,\n",
       " 'human': 7,\n",
       " 'people': 14,\n",
       " 'one': 12,\n",
       " 'started': 18,\n",
       " 'war': 23,\n",
       " 'ai': 1,\n",
       " 'bad': 2,\n",
       " 'thing': 19}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view token vocabulary and counts\n",
    "vect.vocabulary_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `TfidfTransformer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "\n",
    "# initialize tf-idf transformer object\n",
    "transformer = TfidfTransformer(smooth_idf=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TfidfTransformer(norm='l2', smooth_idf=False, sublinear_tf=False,\n",
       "         use_idf=True)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transformer.fit(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a glimpse of how the IDF values look, we are going to print it by placing the IDF values in a python DataFrame. The values will be sorted in ascending order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>idf_weights</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>look</th>\n",
       "      <td>1.916291</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>view</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>twice</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>thing</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>started</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>see</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>second</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>renaissance</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>people</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>part</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>one</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>may</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>matrix</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>least</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>human</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>first</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>definitely</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>change</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>boring</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bad</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ai</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>war</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>watch</th>\n",
       "      <td>2.609438</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             idf_weights\n",
       "look            1.916291\n",
       "2               2.609438\n",
       "view            2.609438\n",
       "twice           2.609438\n",
       "time            2.609438\n",
       "thing           2.609438\n",
       "started         2.609438\n",
       "see             2.609438\n",
       "second          2.609438\n",
       "renaissance     2.609438\n",
       "people          2.609438\n",
       "part            2.609438\n",
       "one             2.609438\n",
       "may             2.609438\n",
       "matrix          2.609438\n",
       "least           2.609438\n",
       "human           2.609438\n",
       "first           2.609438\n",
       "definitely      2.609438\n",
       "change          2.609438\n",
       "boring          2.609438\n",
       "bad             2.609438\n",
       "ai              2.609438\n",
       "war             2.609438\n",
       "watch           2.609438"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df_idf = pd.DataFrame(transformer.idf_, index = vect.get_feature_names(), columns=['idf_weights'] )\n",
    "df_idf.sort_values(by=['idf_weights'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The lower the IDF value of a word, the less unique"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Compute the TFIDF score for  corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use counts from count vectorizer results to compute tf-idf values\n",
    "tfidf = transformer.transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.        , 0.        , 0.        , 0.36419547, 0.        ,\n",
       "        0.        , 0.36419547, 0.        , 0.        , 0.26745392,\n",
       "        0.        , 0.36419547, 0.        , 0.        , 0.        ,\n",
       "        0.36419547, 0.36419547, 0.36419547, 0.        , 0.        ,\n",
       "        0.36419547, 0.        , 0.        , 0.        , 0.        ],\n",
       "       [0.39105193, 0.        , 0.        , 0.        , 0.        ,\n",
       "        0.39105193, 0.        , 0.        , 0.39105193, 0.28717648,\n",
       "        0.        , 0.        , 0.        , 0.39105193, 0.        ,\n",
       "        0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "        0.        , 0.39105193, 0.        , 0.        , 0.39105193],\n",
       "       [0.        , 0.        , 0.        , 0.        , 0.57735027,\n",
       "        0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "        0.57735027, 0.        , 0.        , 0.        , 0.        ,\n",
       "        0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "        0.        , 0.        , 0.57735027, 0.        , 0.        ],\n",
       "       [0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "        0.        , 0.        , 0.4472136 , 0.        , 0.        ,\n",
       "        0.        , 0.        , 0.4472136 , 0.        , 0.4472136 ,\n",
       "        0.        , 0.        , 0.        , 0.4472136 , 0.        ,\n",
       "        0.        , 0.        , 0.        , 0.4472136 , 0.        ],\n",
       "       [0.        , 0.57735027, 0.57735027, 0.        , 0.        ,\n",
       "        0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "        0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "        0.        , 0.        , 0.        , 0.        , 0.57735027,\n",
       "        0.        , 0.        , 0.        , 0.        , 0.        ]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert sparse matrix to numpy array to view\n",
    "tfidf.toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(5, 25)\n"
     ]
    }
   ],
   "source": [
    "print(tfidf.toarray().shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let’s print the tf-idf values of the first document to see if it makes sense. What we are doing below is, placing the tf-idf scores from the first document into a pandas data frame and sorting it in descending order of scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tfidf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>renaissance</th>\n",
       "      <td>0.364195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>boring</th>\n",
       "      <td>0.364195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>first</th>\n",
       "      <td>0.364195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time</th>\n",
       "      <td>0.364195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>may</th>\n",
       "      <td>0.364195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>see</th>\n",
       "      <td>0.364195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>second</th>\n",
       "      <td>0.364195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>look</th>\n",
       "      <td>0.267454</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>war</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>view</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>twice</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>thing</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>started</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>one</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>people</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>part</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ai</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>matrix</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>least</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>human</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>definitely</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>change</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bad</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>watch</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                tfidf\n",
       "renaissance  0.364195\n",
       "boring       0.364195\n",
       "first        0.364195\n",
       "time         0.364195\n",
       "may          0.364195\n",
       "see          0.364195\n",
       "second       0.364195\n",
       "look         0.267454\n",
       "2            0.000000\n",
       "war          0.000000\n",
       "view         0.000000\n",
       "twice        0.000000\n",
       "thing        0.000000\n",
       "started      0.000000\n",
       "one          0.000000\n",
       "people       0.000000\n",
       "part         0.000000\n",
       "ai           0.000000\n",
       "matrix       0.000000\n",
       "least        0.000000\n",
       "human        0.000000\n",
       "definitely   0.000000\n",
       "change       0.000000\n",
       "bad          0.000000\n",
       "watch        0.000000"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(tfidf[0].T.todense(), index=vect.get_feature_names(), columns=[\"tfidf\"])\n",
    "df.sort_values(by=[\"tfidf\"],ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that only certain words have scores. This is because our first document is \"The first time you see The Second Renaissance it may look boring.\" all the words in this document have a tf-idf score and everything else show up as zeroes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the word “ it” is missing from this list. This is possibly due to pre-processing where it removes single and stop characters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The scores above make sense. The more common the word across documents, the lower its score and the more unique a word is to our first document the higher the score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `TfidfVectorizer`\n",
    "`TfidfVectorizer` = `CountVectorizer` + `TfidfTransformer`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are going to use the same 5 documents from above to do the same thing as we did for Tfidftransformer – which is to get the tf-idf scores of a set of documents. But, notice how this is much shorter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "# initialize tf-idf vectorizer object\n",
    "vectorizer = TfidfVectorizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute bag of word counts and tf-idf values\n",
    "X = vectorizer.fit_transform(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.        , 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.30298183, 0.        , 0.        , 0.30298183, 0.        ,\n",
       "       0.        , 0.20291046, 0.        , 0.24444384, 0.        ,\n",
       "       0.30298183, 0.        , 0.        , 0.        , 0.        ,\n",
       "       0.30298183, 0.30298183, 0.30298183, 0.        , 0.40582093,\n",
       "       0.        , 0.30298183, 0.        , 0.        , 0.        ,\n",
       "       0.        , 0.        , 0.        , 0.30298183, 0.        ])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert sparse matrix to numpy array to view\n",
    "X.toarray()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tfidftransformer vs. Tfidfvectorizer\n",
    "\n",
    "In summary, the main difference between the two modules are as follows:\n",
    "\n",
    "* With Tfidftransformer you will systematically compute word counts using CountVectorizer and then compute the Inverse Document Frequency (IDF) values and only then compute the Tf-idf scores.\n",
    "\n",
    "* With Tfidfvectorizer on the contrary, you will do all three steps at once. Under the hood, it computes the word counts, IDF values, and Tf-idf scores all using the same dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}