Newsgroup_Dataset- KNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 20 News Group Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objective :  Parse, Compute Pairwaise Similarity Matrices,  Train and Test using KNN Classification Algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading relevant libraries "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#importing the relevant libraries\n",
    "import numpy as np \n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import time\n",
    "import mnist \n",
    "import collections\n",
    "% matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_20newsgroups\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer,CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#getting the train  & test data\n",
    "newsgroups_train = fetch_20newsgroups(subset='train')\n",
    "newsgroups_test = fetch_20newsgroups(subset = 'test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Selecting the vectorizer "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tfidf_vectorizer = TfidfVectorizer()\n",
    "count_vectorizer = CountVectorizer()\n",
    "train_tfidf_matrix = tfidf_vectorizer.fit_transform(newsgroups_train.data)\n",
    "train_tf_matrix = count_vectorizer.fit_transform(newsgroups_train.data)\n",
    "test_tfidf_matrix = tfidf_vectorizer.transform(newsgroups_test.data)\n",
    "test_tf_matrix = count_vectorizer.transform(newsgroups_test.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(11314, 130107) (7532, 130107)\n"
     ]
    }
   ],
   "source": [
    "print(train_tf_matrix.shape, test_tf_matrix.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(11314, 130107) (7532, 130107)\n"
     ]
    }
   ],
   "source": [
    "print(train_tfidf_matrix.shape, test_tfidf_matrix.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Shape of the matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Shape of tf sparse matrix: (11314, 130107)\n",
      "The Shape of tf idf sparse matrix: (11314, 130107)\n",
      "The Shape of tf sparse matrix: (7532, 130107)\n",
      "The Shape of tf idf sparse matrix: (7532, 130107)\n"
     ]
    }
   ],
   "source": [
    "print(\"The Shape of tf sparse matrix:\",train_tf_matrix.shape)\n",
    "print(\"The Shape of tf idf sparse matrix:\",train_tfidf_matrix.shape)\n",
    "print(\"The Shape of tf sparse matrix:\",test_tf_matrix.shape)\n",
    "print(\"The Shape of tf idf sparse matrix:\",test_tfidf_matrix.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each row represent the documents and the colunms are word indices. \n",
    "\n",
    "In tf matrix the word frequency per documents is stored in matrix. \n",
    "\n",
    "In tfidf matrix the tfidf (tf * idf) value of each word per document is stored in the matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 0 0 0 0 0 0 0 0 0]\n",
      " [0 0 0 0 0 0 0 0 0 0]]\n"
     ]
    }
   ],
   "source": [
    "#lets look at some raw data of the sparse matrix\n",
    "print(train_tf_matrix[0:2,0:10].todense())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "   0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "   0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "   0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "   0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "   0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]]\n"
     ]
    }
   ],
   "source": [
    "#lets look at some raw data of the sparse matrix\n",
    "print(train_tfidf_matrix[50:51,0:100].todense())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sparsity is a major challenge while dealing with text data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cosine Similarity for Train Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#importing the relevant library\n",
    "from sklearn.metrics.pairwise import cosine_similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "The total Running time for computing cosine simlarity of tf matrix in seconds: 19.37595224380493\n"
     ]
    }
   ],
   "source": [
    "#start time \n",
    "start = time.time()\n",
    "\n",
    "#compute the cosine similarity of each doucment with one another\n",
    "ng_tf_cs = cosine_similarity(train_tf_matrix,train_tf_matrix)\n",
    "\n",
    "#end time \n",
    "end = time.time()\n",
    "\n",
    "#total time \n",
    "print(\"\\nThe total Running time for computing cosine simlarity of tf matrix in seconds:\",end-start)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "The total Running time  for computing cosine simlarity of tfidf matrix in seconds: 15.087325096130371\n"
     ]
    }
   ],
   "source": [
    "#start time \n",
    "start = time.time()\n",
    "\n",
    "#compute the cosine similarity of each doucment with one another\n",
    "ng_tfidf_cs = cosine_similarity(train_tfidf_matrix,train_tfidf_matrix)\n",
    "\n",
    "#end time \n",
    "end = time.time()\n",
    "\n",
    "#total time \n",
    "print(\"\\nThe total Running time  for computing cosine simlarity of tfidf matrix in seconds:\",end-start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cosine Similarity for Test Data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "The total Running time for computing cosine simlarity of tf matrix in seconds: 12.637705326080322\n"
     ]
    }
   ],
   "source": [
    "#start time \n",
    "start = time.time()\n",
    "\n",
    "#compute the cosine similarity of each doucment with one another\n",
    "ng_test_tf_cs = cosine_similarity(test_tf_matrix,train_tf_matrix)\n",
    "\n",
    "#end time \n",
    "end = time.time()\n",
    "\n",
    "#total time \n",
    "print(\"\\nThe total Running time for computing cosine simlarity of tf matrix in seconds:\",end-start)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "The total Running time for computing cosine simlarity of tf matrix in seconds: 12.823270082473755\n"
     ]
    }
   ],
   "source": [
    "#start time \n",
    "start = time.time()\n",
    "\n",
    "#compute the cosine similarity of each doucment with one another\n",
    "ng_test_tfidf_cs = cosine_similarity(test_tfidf_matrix,train_tfidf_matrix)\n",
    "\n",
    "#end time \n",
    "end = time.time()\n",
    "\n",
    "#total time \n",
    "print(\"\\nThe total Running time for computing cosine simlarity of tf matrix in seconds:\",end-start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Euclidean Distance for Similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#importing the relevant library\n",
    "from sklearn.metrics.pairwise import euclidean_distances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "The total Running time for computing similarity by euclidean distance of tf matrix in seconds: 20.618422746658325\n"
     ]
    }
   ],
   "source": [
    "#start time \n",
    "start = time.time()\n",
    "\n",
    "#compute the cosine similarity of each doucment with one another\n",
    "ng_tf_ed = euclidean_distances(train_tf_matrix)\n",
    "\n",
    "#end time \n",
    "end = time.time()\n",
    "\n",
    "#total time \n",
    "print(\"\\nThe total Running time for computing similarity by euclidean distance of tf matrix in seconds:\",end-start)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "The total Running time for similarity by euclidean distance of tfidf matrix in seconds: 23.99633550643921\n"
     ]
    }
   ],
   "source": [
    "#start time \n",
    "start = time.time()\n",
    "\n",
    "#compute the cosine similarity of each doucment with one another\n",
    "ng_tfidf_ed = euclidean_distances(train_tfidf_matrix)\n",
    "\n",
    "#end time \n",
    "end = time.time()\n",
    "\n",
    "#total time \n",
    "print(\"\\nThe total Running time for similarity by euclidean distance of tfidf matrix in seconds:\",end-start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Euclidean distances for Test Data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "The total Running time for computing similarity by euclidean distance of tf matrix in seconds: 15.15920352935791\n"
     ]
    }
   ],
   "source": [
    "#start time \n",
    "start = time.time()\n",
    "\n",
    "#compute the cosine similarity of each doucment with one another\n",
    "ng_test_tf_ed = euclidean_distances(test_tf_matrix,train_tf_matrix)\n",
    "\n",
    "#end time \n",
    "end = time.time()\n",
    "\n",
    "#total time \n",
    "print(\"\\nThe total Running time for computing similarity by euclidean distance of tf matrix in seconds:\",end-start)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "The total Running time for similarity by euclidean distance of tfidf matrix in seconds: 14.546274185180664\n"
     ]
    }
   ],
   "source": [
    "#start time \n",
    "start = time.time()\n",
    "\n",
    "#compute the cosine similarity of each doucment with one another\n",
    "ng_test_tfidf_ed = euclidean_distances(test_tfidf_matrix,train_tfidf_matrix)\n",
    "\n",
    "#end time \n",
    "end = time.time()\n",
    "\n",
    "#total time \n",
    "print(\"\\nThe total Running time for similarity by euclidean distance of tfidf matrix in seconds:\",end-start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training accuracy 0.9165635495845855\n"
     ]
    }
   ],
   "source": [
    "#accuracy on training data using cosine similarity and tfidf vector\n",
    "k = 5\n",
    "sum = 0\n",
    "for i in range(0,len(newsgroups_train.data)):\n",
    "    similar_index =  np.argsort(ng_tfidf_cs[i])[:-(k+1):-1].tolist()\n",
    "    l = newsgroups_train.target[similar_index].tolist()\n",
    "    label = max(l,key=l.count)\n",
    "    actual_label = newsgroups_train.target[i]\n",
    "    if label == actual_label:\n",
    "        sum += 1\n",
    "print(\"training accuracy\",sum/len(newsgroups_train.data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training accuracy 0.8680395969595192\n"
     ]
    }
   ],
   "source": [
    "#accuracy on training data using cosine similarity and tfidf vector\n",
    "k = 10\n",
    "sum = 0\n",
    "for i in range(0,len(newsgroups_train.data)):\n",
    "    similar_index =  np.argsort(ng_tfidf_cs[i])[:-(k+1):-1].tolist()\n",
    "    l = newsgroups_train.target[similar_index].tolist()\n",
    "    label = max(l,key=l.count)\n",
    "    actual_label = newsgroups_train.target[i]\n",
    "    if label == actual_label:\n",
    "        sum += 1\n",
    "print(\"training accuracy\",sum/len(newsgroups_train.data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training accuracy 0.8656531730599257\n"
     ]
    }
   ],
   "source": [
    "#accuracy on training data using cosine similarity and tf vector\n",
    "k = 5\n",
    "sum = 0\n",
    "for i in range(0,len(newsgroups_train.data)):\n",
    "    similar_index =  np.argsort(ng_tf_cs[i])[:-(k+1):-1].tolist()\n",
    "    l = newsgroups_train.target[similar_index].tolist()\n",
    "    label = max(l,key=l.count)\n",
    "    actual_label = newsgroups_train.target[i]\n",
    "    if label == actual_label:\n",
    "        sum += 1\n",
    "print(\"training accuracy\",sum/len(newsgroups_train.data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training accuracy 0.9165635495845855\n"
     ]
    }
   ],
   "source": [
    "#accuracy on training data using euclidean distances and tfidf vector\n",
    "k = 5\n",
    "sum = 0\n",
    "for i in range(0,len(newsgroups_train.data)):\n",
    "    similar_index =  np.argsort(ng_tfidf_ed[i])[:k].tolist()\n",
    "    l = newsgroups_train.target[similar_index].tolist()\n",
    "    label = max(l,key=l.count)\n",
    "    actual_label = newsgroups_train.target[i]\n",
    "    if label == actual_label:\n",
    "        sum += 1\n",
    "print(\"training accuracy\",sum/len(newsgroups_train.data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training accuracy 0.8793530139649991\n"
     ]
    }
   ],
   "source": [
    "#accuracy on training data using euclidean distances and tf vector\n",
    "k = 5\n",
    "sum = 0\n",
    "for i in range(0,len(newsgroups_train.data)):\n",
    "    similar_index =  np.argsort(ng_tf_ed[i])[:k].tolist()\n",
    "    l = newsgroups_train.target[similar_index].tolist()\n",
    "    label = max(l,key=l.count)\n",
    "    actual_label = newsgroups_train.target[i]\n",
    "    if label == actual_label:\n",
    "        sum += 1\n",
    "print(\"training accuracy\",sum/len(newsgroups_train.data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training accuracy 0.6755177907594264\n"
     ]
    }
   ],
   "source": [
    "#accuracy on test data using cosine similarity and tfidf vector\n",
    "k = 5\n",
    "sum = 0\n",
    "for i in range(0,len(newsgroups_test.data)):\n",
    "    similar_index =  np.argsort(ng_test_tfidf_cs[i])[:-(k+1):-1].tolist()\n",
    "    l = newsgroups_train.target[similar_index].tolist()\n",
    "    label = max(l,key=l.count)\n",
    "    actual_label = newsgroups_test.target[i]\n",
    "    if label == actual_label:\n",
    "        sum += 1\n",
    "print(\"training accuracy\",sum/len(newsgroups_test.data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training accuracy 0.4305629314922995\n"
     ]
    }
   ],
   "source": [
    "#accuracy on test data using cosine similarity and tf vector\n",
    "k = 5\n",
    "sum = 0\n",
    "for i in range(0,len(newsgroups_test.data)):\n",
    "    similar_index =  np.argsort(ng_test_tf_cs[i])[:-(k+1):-1].tolist()\n",
    "    l = newsgroups_train.target[similar_index].tolist()\n",
    "    label = max(l,key=l.count)\n",
    "    actual_label = newsgroups_test.target[i]\n",
    "    if label == actual_label:\n",
    "        sum += 1\n",
    "print(\"training accuracy\",sum/len(newsgroups_test.data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training accuracy 0.6755177907594264\n"
     ]
    }
   ],
   "source": [
    "#accuracy on test data using euclidean distances and tfidf vector\n",
    "k = 5\n",
    "sum = 0\n",
    "for i in range(0,len(newsgroups_test.data)):\n",
    "    similar_index =  np.argsort(ng_test_tfidf_ed[i])[:k].tolist()\n",
    "    l = newsgroups_train.target[similar_index].tolist()\n",
    "    label = max(l,key=l.count)\n",
    "    actual_label = newsgroups_test.target[i]\n",
    "    if label == actual_label:\n",
    "        sum += 1\n",
    "print(\"training accuracy\",sum/len(newsgroups_test.data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training accuracy 0.38382899628252787\n"
     ]
    }
   ],
   "source": [
    "#accuracy on test data using euclidean distances and tf vector\n",
    "k = 5\n",
    "sum = 0\n",
    "for i in range(0,len(newsgroups_test.data)):\n",
    "    similar_index =  np.argsort(ng_test_tf_ed[i])[:k].tolist()\n",
    "    l = newsgroups_train.target[similar_index].tolist()\n",
    "    label = max(l,key=l.count)\n",
    "    actual_label = newsgroups_test.target[i]\n",
    "    if label == actual_label:\n",
    "        sum += 1\n",
    "print(\"training accuracy\",sum/len(newsgroups_test.data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[8665, 4772, 9080, 2931, 4495]\n",
      "[2, 2, 2, 2, 2]\n",
      "comp.os.ms-windows.misc\n",
      "comp.os.ms-windows.misc\n",
      "comp.os.ms-windows.misc\n",
      "comp.os.ms-windows.misc\n",
      "comp.os.ms-windows.misc\n",
      "2\n",
      "4\n"
     ]
    }
   ],
   "source": [
    "#Print labels\n",
    "similar_index = np.argsort(ng_test_tf_cs[123])[:5].tolist()\n",
    "print(similar_index)\n",
    "l = newsgroups_train.target[similar_index].tolist()\n",
    "print(l)\n",
    "for i in l :\n",
    "    print(newsgroups_train.target_names[i])\n",
    "label = max(l,key=l.count)\n",
    "print(label)\n",
    "actual_label = newsgroups_train.target[i]\n",
    "print(actual_label)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}