report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part I: Theory Questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MLE \n",
    "### Answer 1\n",
    "Given observations $x _ { 1 } , \\ldots , x _ { N } ,$ the likelihood of those observations for a certain $\\mu$ and $\\sigma ^ { 2 }$ (assuming\n",
    "that the observations came from a Gaussian distribution) is\n",
    "First, we derive the likelihood term:\n",
    "$$p \\left( x _ { 1 } , \\ldots , x _ { N } | \\mu , \\sigma ^ { 2 } \\right) = \\prod _ { n = 1 } ^ { N } \\frac { 1 } { \\sqrt { 2 \\pi } \\sigma } \\exp \\left\\{ \\frac { - \\left( x _ { n } - \\mu \\right) ^ { 2 } } { 2 \\sigma ^ { 2 } } \\right\\}$$\n",
    "\n",
    "and the log likelihood is;\n",
    "\n",
    "$$\\mathcal { L } ( \\mu , \\sigma ) = - \\frac { 1 } { 2 } N \\log \\left( 2 \\pi \\sigma ^ { 2 } \\right) - \\sum _ { n = 1 } ^ { N } \\frac { \\left( x _ { n } - \\mu \\right) ^ { 2 } } { 2 \\sigma ^ { 2 } }$$\n",
    "\n",
    "We can then find the values of $\\mu$ and $\\sigma ^ { 2 }$ that maximize the log likelihood by taking derivative with respect to the desired variable and solving the equation obtained. By doing so,\n",
    "we find that the MLE of the mean is ;\n",
    "\n",
    "$$\\hat { \\mu } = \\frac { 1 } { N } \\sum _ { n = 1 } ^ { N } x _ { n }$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Answer 2\n",
    "The other probability, $p \\left( x _ { i } = 0 | c \\right)$ is given by the normalisation requirement,\n",
    "$p \\left( x _ { i } = 0 | c \\right) = 1 - p \\left( x _ { i } = 1 | c \\right) = 1 - \\theta _ { i } ^ { c }$\n",
    "\n",
    "Based on the NB conditional independence assumption the probability of observing a vector $x$ can be\n",
    "compactly written\n",
    "\n",
    "$p ( \\mathbf { x } | c ) = \\prod _ { i = 1 } ^ { D } p \\left( x _ { i } | c \\right) = \\prod _ { i = 1 } ^ { D } \\left( \\theta _ { i } ^ { c } \\right) ^ { x _ { i } } \\left( 1 - \\theta _ { i } ^ { c } \\right) ^ { 1 - x _ { i } }$\n",
    "\n",
    "In the above expression, $x _ { i }$ is either 0 or 1 and hence each $i$ term contributes a factor $\\theta _ { i } ^ { c }$ if $x _ { i } = 1$ or $1 - \\theta _ { i } ^ { c }$\n",
    "if $x _ { i } = 0 .$ Together with the assumption that the training data is i.i.d. generated, the log likelihood of the\n",
    "attributes and class labels labels labels is\n",
    "\n",
    "$\\begin{array} { r l } { L } & { = \\sum _ { n } \\log p \\left( \\mathbf { x } ^ { n } , c ^ { n } \\right) = \\sum _ { n } \\log p \\left( c ^ { n } \\right) \\prod _ { i } p \\left( x _ { i } ^ { n } | c ^ { n } \\right) } \\\\ { } & { = \\sum _ { i , n } x _ { i } ^ { n } \\log \\theta _ { i } ^ { c ^ { n } } + \\left( 1 - x _ { i } ^ { n } \\right) \\log \\left( 1 - \\theta _ { i } ^ { c ^ { n } } \\right) + n _ { 0 } \\log p ( c = 0 ) + n _ { 1 } \\log p ( c = 1 ) } \\end{array}$\n",
    "\n",
    "We can find the Maximum Likelihood optimal $\\theta _ { i } ^ { c }$ by differentiating $w . r . t . \\theta _ { i } ^ { c }$ and equating to zero, giving\n",
    "\n",
    "$\\begin{aligned} \\theta _ { i } ^ { c } = p \\left( x _ { i } = 1 | c \\right) & = \\frac { \\sum _ { n } \\mathbb { I } \\left[ x _ { i } ^ { n } = 1 , c ^ { n } = c \\right] } { \\sum _ { n } \\mathbb { I } \\left[ x _ { i } ^ { n } = 0 , c ^ { n } = c \\right] + \\mathbb { I } \\left[ x _ { i } ^ { n } = 1 , c ^ { n } = c \\right] } \\\\ & = \\frac { \\text { number of times } x _ { i } = 1 \\text { for class } c } { \\text { number of datapoints in class } c } \\end{aligned}$\n",
    "\n",
    "\n",
    "\n",
    "$p ( c ) = \\frac { \\text { number of times class c occurs } } { \\text { total number of data points } }$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Answer 3\n",
    "\n",
    "Since the sample is (3,0,2,1,3,2,1,0,2,1), the likelihood is\n",
    "\n",
    "$$\\begin{aligned} L ( \\theta ) = & P ( X = 3 ) P ( X = 0 ) P ( X = 2 ) P ( X = 1 ) P ( X = 3 )  \\times P ( X = 2 ) P ( X = 1 ) P ( X = 0 ) P ( X = 2 ) P ( X = 1 ) \\end{aligned}$$\n",
    "\n",
    "Substituting from the probability distribution given above, we have\n",
    "\n",
    "$$L ( \\theta ) = \\prod _ { i = 1 } ^ { n } P \\left( X _ { i } | \\theta \\right) = \\left( \\frac { 2 \\theta } { 3 } \\right) ^ { 2 } \\left( \\frac { \\theta } { 3 } \\right) ^ { 3 } \\left( \\frac { 2 ( 1 - \\theta ) } { 3 } \\right) ^ { 3 } \\left( \\frac { 1 - \\theta } { 3 } \\right) ^ { 2 }$$\n",
    "Clearly, the likelihood function $L(θ)$ is not easy to maximize.Let us look at the log likelihood function\n",
    "\n",
    "$$\\begin{array} { r l } { l ( \\theta ) } & { = \\log L ( \\theta ) = \\sum _ { i = 1 } ^ { n } \\log P \\left( X _ { i } | \\theta \\right) } \\\\ { } & { = 2 \\left( \\log \\frac { 2 } { 3 } + \\log \\theta \\right) + 3 \\left( \\log \\frac { 1 } { 3 } + \\log \\theta \\right) + 3 \\left( \\log \\frac { 2 } { 3 } + \\log ( 1 - \\theta ) \\right) + 2 \\left( \\log \\frac { 1 } { 3 } + \\log ( 1 - \\theta ) \\right) } \\\\ { } & { = C + 5 \\log \\theta + 5 \\log ( 1 - \\theta ) } \\end{array}$$\n",
    "\n",
    "where $C$ is a constant which does not depend on $\\theta$ . It can be seen that the log likelihood\n",
    "function is easier to maximize compared to the likelihood function.\n",
    "Let the derivative of $l ( \\theta )$ with respect to $\\theta$ be zero:\n",
    "\n",
    "$$\\frac { d l ( \\theta ) } { d \\theta } = \\frac { 5 } { \\theta } - \\frac { 5 } { 1 - \\theta } = 0$$\n",
    "and the solution gives us the MLE, which is $\\hat { \\theta } = 0.5 .$ We remember that the method of\n",
    "moment estimation is $\\hat { \\theta } = 5 / 12 ,$ which is different from MLE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naïve Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The conditional probabiltys with laplace smoothing;\n",
    "\n",
    "\n",
    "\n",
    "$p ( \\mathbf { rich } | content ) = \\frac {3+1}{18}$ ,$p ( \\mathbf { not rich } | content ) = \\frac {1+1}{18}$ \n",
    "\n",
    "$p ( \\mathbf { married } | content ) = \\frac {2+1}{18}$ ,$p ( \\mathbf { unmarried } | content ) = \\frac {2+1}{18}$ \n",
    "\n",
    "$p ( \\mathbf { helathy } | content ) = \\frac {3+1}{18}$ ,$p ( \\mathbf { unhealthy } | content ) = \\frac {1+1}{18}$\n",
    "\n",
    "$p ( \\mathbf { rich } | notcontent ) = \\frac {4+1}{21}$ ,$p ( \\mathbf { not rich } | notcontent ) = \\frac {1+1}{21}$ \n",
    "\n",
    "$p ( \\mathbf { married } | notcontent ) = \\frac {1+1}{21}$ ,$p ( \\mathbf { unmarried } | notcontent ) = \\frac {4+1}{21}$ \n",
    "\n",
    "$p ( \\mathbf { helathy } | notcontent ) = \\frac {1+1}{21}$ ,$p ( \\mathbf { unhealthy } | notcontent ) = \\frac {4+1}{21}$\n",
    "\n",
    "\n",
    "\n",
    "$p ( \\mathbf { content })= \\frac {4}{9}$  $p ( \\mathbf { notcontent })=\\frac {5}{9}$\n",
    "\n",
    "### 1)\n",
    "\n",
    "test= (0,1,1)\n",
    "\n",
    "naive bayes =  $ \\frac {4}{9} x \\frac {2}{18}  x \\frac {3}{18} x \\frac {4}{18} = 0,0018$\n",
    "\n",
    "### 2)\n",
    "test = (0,1,x)\n",
    "\n",
    " $ \\frac {4}{9} x \\frac {2}{18}  x \\frac {3}{18} x \\frac {4}{18} + \\frac {4}{9} x \\frac {2}{18}  x \\frac {3}{18} x \\frac {2}{18} = 0,0027$ \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PART II: Detection of Fake News"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##   INTRODUCTIONS\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Classifiers are machine learning models that can take a set of examples and learn which class each example falls into. You can then ask it to classify content that it has never seen before, with impressive accuracy. We will try to predict whether new news shares are real by using fake-real training data in this project.\n",
    "\n",
    "\n",
    "Naïve Bayes is a simple, yet effective and commonly-used, machine learning classifier. Naïve Bayes classifiers have been especially popular for text classification, and are a traditional solution for problems such as fake-real detection.So we will try to make classification using Naïve Bayes classifier.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naïve Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Naïve Bayes classification provides data that is taught to the system at a certain rate (eg 100 pieces). The data submitted for teaching must have a class / category. With the probabilistic operations performed on the taught data, the new test data presented to the system is operated according to the previously obtained probability values and it is tried to determine which category of test data is given. Of course, the greater the number of data taught, the more precise it is to determine the actual category of test data.\n",
    "\n",
    "The Naïve Bayes classification methodology can have many uses, but it is important how it is classified here rather than what it is classified. In other words, the data to be taught can be binary or text data, where it is important to establish a proportional relationship between these data rather than the type of data and what it is."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a multinomial event model, samples (feature vectors) represent the frequencies with which certain events have been generated by a multinomial ${\\displaystyle (p_{1},\\dots ,p_{n})}$ where ${\\displaystyle p_{i}}$ is the probability that event i occurs (or $K$ such multinomials in the multiclass case). A feature vector ${\\displaystyle \\mathbf {x} =(x_{1},\\dots ,x_{n})}$ is then a histogram, with ${\\displaystyle x_{i}}$ counting the number of times event $i$ was observed in a particular instance. This is the event model typically used for document classification, with events representing the occurrence of a word in a single document (e.g Bag Of Words assumption). The likelihood of observing a histogram $x$ is given by\n",
    "\n",
    "${\\displaystyle p(\\mathbf {x} \\mid C_{k})={\\frac {(\\sum _{i}x_{i})!}{\\prod _{i}x_{i}!}}\\prod _{i}{p_{ki}}^{x_{i}}}$\n",
    "\n",
    "The multinomial naive Bayes classifier becomes a linear classifier when expressed in log-space:[2]\n",
    "\n",
    "${\\displaystyle {\\begin{aligned}\\log p(C_{k}\\mid \\mathbf {x} )&\\varpropto \\log \\left(p(C_{k})\\prod _{i=1}^{n}{p_{ki}}^{x_{i}}\\right)\\\\&=\\log p(C_{k})+\\sum _{i=1}^{n}x_{i}\\cdot \\log p_{ki}\\\\&=b+\\mathbf {w} _{k}^{\\top }\\mathbf {x} \\end{aligned}}}$\n",
    "\n",
    "where ${\\displaystyle b=\\log p(C_{k})}$ and ${\\displaystyle w_{ki}=\\log p_{ki}}$\n",
    "\n",
    "If a given class and feature value never occur together in the training data, then the frequency-based probability estimate will be zero. This is problematic because it will wipe out all information in the other probabilities when they are multiplied. Therefore, it is often desirable to incorporate a small-sample correction, called pseudocount, in all probability estimates such that no probability is ever set to be exactly zero. This way of regularizing naive Bayes is called Laplace smoothing when the pseudocount is one.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Laplace Smoothing in Naïve Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $c$ refer to a class (such as Positive or Negative), and let $w$ refer to a token or word.\n",
    "\n",
    "The maximum likelihood estimator for $P(w|c)$ is\n",
    "\n",
    "$$   \\frac{count(w,c)}{count(c)}= \\frac{counts\\,w\\,in\\,class\\,c}{counts\\,of\\,words\\,in\\,class\\,c}$$\n",
    "\n",
    "\n",
    "This estimation of $P(w|c)$ could be problematic since it would give us probability $0$ for documents with unknown words. A common way of solving this problem is to use Laplace smoothing.\n",
    "\n",
    "Let $V$ be the set of words in the training set, add a new element $UNK$ (for unknown) to the set of words.\n",
    "\n",
    "Define\n",
    "\n",
    "$$   P(w|c) = \\frac{count(w,c)+1}{count(c)+|V|+1}$$\n",
    "\n",
    "where $V$ refers to the vocabulary (the words in the training set).\n",
    "\n",
    "In particular, any unknown word will have probability\n",
    "\n",
    "$$   P(w|c) = \\frac{1}{count(c)+|V|+1}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding The Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the training data provided we try to identify specific words for real and false news in this section.Instead of using the highest frequency words found in the classes, I thought it was more logical to use the first three of the words in their class but not in the other class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3 Specific keys for detect real news:\n",
      "  Word/WordPairs  Frequency\n",
      "0          korea         62\n",
      "1       turnbull         48\n",
      "2         travel         47\n",
      "\n",
      "\n",
      " 3 Specific keys for detect fake news:\n",
      "  Word/WordPairs  Frequency\n",
      "0       breaking         24\n",
      "1          soros         18\n",
      "2          woman         13\n"
     ]
    }
   ],
   "source": [
    "import main\n",
    "main.main(1,PrintCommand=\"specific\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementing Naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, I used the multinomal naive bayes classifier method to keep the words in the bag of words structure.My algorithm designed with N-gram Based Model.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What is the N-gram Based Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "N-gram modeling is a popular feature identification and analysis approach used in\n",
    "language modeling and natural language processing fields.\n",
    "\n",
    "N-gram is a contiguous sequence of items with length n. It could be a sequence of\n",
    "words, bytes, syllables, or characters. The most used n-gram models in text categorization\n",
    "are word-based and character-based n-grams. Examples of n-gram models commonly\n",
    "used include unigram (n=1), bigram (n=2),etc.\n",
    "\n",
    "When building an n-gram based classifier, the size n is usually a fixed number\n",
    "throughout the whole corpus. The unigrams are commonly known as “the bag of words”\n",
    "model. The bag of words model does not take into consideration the order of the phrase\n",
    "in contrast to a higher order n-gram model. The n-gram model is one of the basic and\n",
    "efficient models for text categorization and language processing. It allows automatic\n",
    "capture of the most frequent words in the corpus; it can be applied to any language since\n",
    "it does not need segmentation of the text in words. Furthermore, it is flexible against\n",
    "spelling mistake and deformations since it recognizes particles of the phrase/words.\n",
    "\n",
    "In this Assignment, we will be using word-based n-gram model to represent the context\n",
    "of the document and generate features to classify the document. One of the goals of this\n",
    "assignment is to develop a simple n-gram based classifier to differentiate between fake and real\n",
    "opinions.The idea is to generate various sets of n-gram frequency profiles from the\n",
    "training data to represent fake and truthful opinions. We used two values of n to\n",
    "generate and extract the n-gram features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Unigram "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we used the unigram structure, we kept the words singularly in the structure of the bag of words and we tried to determine the class of the new data by calculating the $multinomal\\,naive\\,bayes$ probabilities according to the document frequencies of these words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The application results are below ;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Correct classified news: 422\n",
      "Steamming : False\n",
      "TF-IDF : False\n",
      "The occurrences of words : 1\n",
      "Stopwords: None\n",
      "Accuracy:  86.29856850715747\n",
      "####################################\n"
     ]
    }
   ],
   "source": [
    "main.main(1,PrintCommand=\"General Results\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bigram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we used the unigram structure, we kept the words pairs in the structure of the bag of words and we tried to determine the class of the new data by calculating the multinomal naive bayes probabilities according to the document frequencies of these words.\n",
    "\n",
    "While this method is applied, $token$ is added since the probability of words being the beginning of the sentence or the end of the sentence is also important."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The application results are below ;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Correct classified news: 418\n",
      "Steamming : False\n",
      "TF-IDF : False\n",
      "The occurrences of words : 2\n",
      "Stopwords: None\n",
      "Accuracy:  85.48057259713701\n",
      "####################################\n"
     ]
    }
   ],
   "source": [
    "main.main(2,PrintCommand=\"General Results\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyzing effect of the words on prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section I applied $TF-IDF$ to normalize the word frequencies. So i could decide which words were more important for the document.Then I listed which words' presences and absences could be effective for the classification.\n",
    "\n",
    "In this section, I did the only analysis for the binary situations because the absence words in the false news that strengthened the possibility real news, likewise other situations include cross possibility.\n",
    "\n",
    "Repeating my implementation for unigram and bigram."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.In Unigram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "10 words whose presence most strongly predicts that the news is real.\n",
      "And whose absence most strongly predicts that the news is fake.\n",
      "\n",
      "  Word/WordPairs  Frequency\n",
      "0          korea  17.346456\n",
      "1         travel  14.625705\n",
      "2       turnbull  14.284775\n",
      "3      australia  10.034324\n",
      "4        climate   7.774017\n",
      "5          paris   6.784150\n",
      "6        refugee   6.766278\n",
      "7         debate   5.928145\n",
      "8           asia   5.711946\n",
      "9          flynn   5.455626\n"
     ]
    }
   ],
   "source": [
    "main.main(1,tfidf=True,PrintCommand=\"presence\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "10 words whose absence most strongly predicts that the news is real.\n",
      "And whose presence most strongly predicts that the news is fake.\n",
      "\n",
      "  Word/WordPairs  Frequency\n",
      "0       breaking   6.244817\n",
      "1          soros   4.452072\n",
      "2          woman   3.473386\n",
      "3          steal   3.381297\n",
      "4           duke   3.132043\n",
      "5         reason   3.057646\n",
      "6      interview   2.849209\n",
      "7             dr   2.819662\n",
      "8       homeless   2.798578\n",
      "9             my   2.732481\n"
     ]
    }
   ],
   "source": [
    "main.main(1,tfidf=True,PrintCommand=\"absence\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.In Bigram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "10 words whose presence most strongly predicts that the news is real.\n",
      "And whose absence most strongly predicts that the news is fake.\n",
      "\n",
      "     Word/WordPairs  Frequency\n",
      "0       north korea  12.453395\n",
      "1        travel ban   9.930787\n",
      "2         ban _eos_   7.058293\n",
      "3       korea _eos_   6.061345\n",
      "4      _s_ turnbull   4.586553\n",
      "5      trump travel   4.498065\n",
      "6  malcolm turnbull   3.964054\n",
      "7     trumps travel   3.827668\n",
      "8       james comey   3.685592\n",
      "9    comments _eos_   3.495364\n"
     ]
    }
   ],
   "source": [
    "main.main(2,tfidf=True,PrintCommand=\"presence\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "10 words whose absence most strongly predicts that the news is real.\n",
      "And whose presence most strongly predicts that the news is fake.\n",
      "\n",
      "   Word/WordPairs  Frequency\n",
      "0       _s_ watch   5.008082\n",
      "1     _s_ comment   4.463473\n",
      "2    _s_ breaking   4.233264\n",
      "3       trump won   2.782663\n",
      "4      daily wire   2.602999\n",
      "5      wire _eos_   2.602999\n",
      "6      voting for   2.569260\n",
      "7        will win   2.294462\n",
      "8       fame star   2.266078\n",
      "9  breaking trump   2.144844\n"
     ]
    }
   ],
   "source": [
    "main.main(2,tfidf=True,PrintCommand=\"absence\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### StopWords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stopwords are insignificant words in a language that will create noise when used as\n",
    "features in text classification. These are words commonly used in a lot sentences to help\n",
    "connect thought or to assist in the sentence structure. Articles, prepositions and\n",
    "conjunctions and some pronouns are considered stop words. We removed common words\n",
    "such as, a, about, an, are, as, at, be, by, for, from, how, in, is, of, on, or, that, the, these,\n",
    "this, too, was, what, when, where, who, will, etc. Those words were removed from each\n",
    "document. And classification successes re-evaluated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.In Unigram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "10 words whose presence most strongly predicts that the news is real.\n",
      "And whose absence most strongly predicts that the news is fake.\n",
      "\n",
      "  Word/WordPairs  Frequency\n",
      "0          korea  19.479725\n",
      "1       turnbull  15.795719\n",
      "2         travel  15.778778\n",
      "3      australia  11.153182\n",
      "4        climate   8.179058\n",
      "5        refugee   7.159126\n",
      "6          paris   7.103846\n",
      "7         debate   6.317261\n",
      "8           asia   6.135109\n",
      "9       congress   5.973006\n"
     ]
    }
   ],
   "source": [
    "main.main(1,stopWords=\"english\",tfidf=True,PrintCommand=\"presence\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "10 words whose absence most strongly predicts that the news is real.\n",
      "And whose presence most strongly predicts that the news is fake.\n",
      "\n",
      "  Word/WordPairs  Frequency\n",
      "0       breaking   6.750180\n",
      "1          soros   4.805717\n",
      "2          steal   3.967294\n",
      "3          woman   3.854602\n",
      "4         reason   3.508850\n",
      "5           duke   3.389549\n",
      "6      interview   3.158593\n",
      "7             dr   3.100562\n",
      "8       homeless   2.946114\n",
      "9      landslide   2.903458\n"
     ]
    }
   ],
   "source": [
    "main.main(1,stopWords=\"english\",tfidf=True,PrintCommand=\"absence\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.In Bigram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "10 words whose presence most strongly predicts that the news is real.\n",
      "And whose absence most strongly predicts that the news is fake.\n",
      "\n",
      "     Word/WordPairs  Frequency\n",
      "0       north korea  14.366889\n",
      "1        travel ban  10.909364\n",
      "2         ban _eos_   7.634519\n",
      "3       korea _eos_   7.003623\n",
      "4      _s_ turnbull   5.723430\n",
      "5      trump travel   4.941856\n",
      "6  malcolm turnbull   4.455249\n",
      "7     trumps travel   4.272293\n",
      "8    comments _eos_   4.210005\n",
      "9   australia _eos_   4.059922\n"
     ]
    }
   ],
   "source": [
    "main.main(2,stopWords=\"english\",tfidf=True,PrintCommand=\"presence\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "10 words whose absence most strongly predicts that the news is real.\n",
      "And whose presence most strongly predicts that the news is fake.\n",
      "\n",
      "   Word/WordPairs  Frequency\n",
      "0       _s_ watch   6.038749\n",
      "1     _s_ comment   5.572620\n",
      "2    _s_ breaking   4.798081\n",
      "3       trump won   4.240658\n",
      "4      daily wire   2.875408\n",
      "5      wire _eos_   2.875408\n",
      "6       fame star   2.572662\n",
      "7  breaking trump   2.410352\n",
      "8    george soros   2.364295\n",
      "9         _s_ cnn   2.272511\n"
     ]
    }
   ],
   "source": [
    "main.main(2,stopWords=\"english\",tfidf=True,PrintCommand=\"absence\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### StopWords Removal Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally stopwords are words that are likely to go through a large number of all classes.Therefore it is illogical to include it in the words to be considered for classification.But this situation may vary according to the classified content.For example, when classifying using unigram, it may increase the success of classification. But some words can make different attributes with stopwords while using bigram and this stuation may decrease classification success. \n",
    "\n",
    "For the classification we made, it may vary according to the given training data. For example;\n",
    "        \n",
    "        The word \"to\" may be used in real news, but may never be used in fake news.\n",
    "        In this case the new news may contain a large frequency of words \"to\" and this may raise the probability of real news classification. But when we remove this word, the probability of being sent to the fake class may increase.\n",
    "        As another example of \"to trump\" may increase the probability of fake class. When we remove the word \"to\", the probability can be greatly reduced. This is also a large number of situations when using the bigram.\n",
    "        \n",
    "When this type of situation is taken into consideration, the result is negative for the training data we use. so I think it's not logical to use stopwords."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After tokenizing the data, the next step is to transform the tokens into a standard form.\n",
    "Stemming, simply, is changing the words into their original form, and decreasing the\n",
    "number of word types or classes in the data. For example, the words “Running,” ”Ran”\n",
    "and “Runner” will be reduced to the word “run.” We use stemming to make classification\n",
    "faster and efficient.\n",
    "\n",
    "This may affect the success rate according to the given data as in stopwords. For this classification, I can say that this is not the right method.Only increases when bigram is used, but the same success cannot be achieved when using unigram."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Results & Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When I make classification for the given test data, the results obtained for all cases are below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    N_gram Stop words   Stem  TF-IDF  Correct classified   Accuracy\n",
      "0        1       None  False   False                 422  86.298569\n",
      "1        1       None  False    True                 420  85.889571\n",
      "2        1       None   True   False                 414  84.662577\n",
      "3        1       None   True    True                 409  83.640082\n",
      "4        1    english  False   False                 412  84.253579\n",
      "5        1    english  False    True                 409  83.640082\n",
      "6        1    english   True   False                 396  80.981595\n",
      "7        1    english   True    True                 401  82.004090\n",
      "8        2       None  False   False                 418  85.480573\n",
      "9        2       None  False    True                 417  85.276074\n",
      "10       2       None   True   False                 421  86.094070\n",
      "11       2       None   True    True                 414  84.662577\n",
      "12       2    english  False   False                 396  80.981595\n",
      "13       2    english  False    True                 395  80.777096\n",
      "14       2    english   True   False                 401  82.004090\n",
      "15       2    english   True    True                 396  80.981595\n"
     ]
    }
   ],
   "source": [
    "main.All_Results(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown by the results, it is seen that there are two most suitable methods according to the given data after applying all the effects. Using unigram while using the data in a lean form is one of them. The other one is to use the bigram with stemming words. In other cases the classification success was adversely affected.\n",
    "\n",
    "As a result of our operations, by using naive bayes when making classification based on words, by applying various operations on words, we have taught that different results can be obtained when we shape similarity situations according to different criteria, different meanings can be obtained by looking at the relationship of words with each other.According to this information learned in some cases when the classification is done better in some cases has been determined that worse classifications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##                          $$\\\\Muhammed\\,Enes\\\\KOÇAK\\\\21427119$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}