04机器学习实战-第4章 贝叶斯分类.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#第四章-贝叶斯分类\" data-toc-modified-id=\"第四章-贝叶斯分类-1\">第四章 贝叶斯分类</a></span><ul class=\"toc-item\"><li><span><a href=\"#本章内容\" data-toc-modified-id=\"本章内容-1.1\">本章内容</a></span></li><li><span><a href=\"#朴素贝叶斯\" data-toc-modified-id=\"朴素贝叶斯-1.2\">朴素贝叶斯</a></span></li><li><span><a href=\"#摘要\" data-toc-modified-id=\"摘要-1.3\">摘要</a></span><ul class=\"toc-item\"><li><span><a href=\"#贝叶斯定理\" data-toc-modified-id=\"贝叶斯定理-1.3.1\">贝叶斯定理</a></span></li></ul></li><li><span><a href=\"#使用条件概率来分类\" data-toc-modified-id=\"使用条件概率来分类-1.4\">使用条件概率来分类</a></span></li><li><span><a href=\"#使用Python进行文本分类\" data-toc-modified-id=\"使用Python进行文本分类-1.5\">使用Python进行文本分类</a></span><ul class=\"toc-item\"><li><span><a href=\"#准备数据：从文本中构建词向量\" data-toc-modified-id=\"准备数据：从文本中构建词向量-1.5.1\">准备数据：从文本中构建词向量</a></span></li><li><span><a href=\"#训练算法：从词向量计算频率\" data-toc-modified-id=\"训练算法：从词向量计算频率-1.5.2\">训练算法：从词向量计算频率</a></span></li><li><span><a href=\"#朴素贝叶斯分类\" data-toc-modified-id=\"朴素贝叶斯分类-1.5.3\">朴素贝叶斯分类</a></span></li></ul></li><li><span><a href=\"#半朴素贝叶斯分类\" data-toc-modified-id=\"半朴素贝叶斯分类-1.6\">半朴素贝叶斯分类</a></span></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第四章 贝叶斯分类\n",
    "\n",
    "## 本章内容\n",
    "> 1. 使用概率分布进行分类  \n",
    "2. 学习朴素贝叶斯分类器  \n",
    "3. 解析RSS源数据  \n",
    "4. 使用朴素贝叶斯来分析不同地区的态度\n",
    "\n",
    "## 朴素贝叶斯\n",
    "\n",
    ">优点：在数据较少的情况下仍然有效，可以处理多类别问题。  \n",
    "缺点：对于输入数据的准备方式较为敏感。  \n",
    "适用数据类型：标称型数据。  \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## 摘要  \n",
    "\n",
    "1. 贝叶斯定理   \n",
    "2. 朴素贝叶斯分类  \n",
    "3. 半朴素贝叶斯分类  \n",
    "4. EM算法  \n",
    "5. 代码实现与注释  \n",
    "\n",
    "\n",
    "    \n",
    "### 贝叶斯定理\n",
    "\n",
    "贝叶斯定理公式\n",
    "\n",
    "$$P(A|B) = \\frac{P(A)P(B|A)}{P(B)}$$\n",
    "\n",
    "我个人觉得，贝叶斯公式其实就是条件概率的推广，每次想到贝叶斯公式，我就会在脑中冒出下面一幅图：  \n",
    "\n",
    "<img src=\"https://pic4.zhimg.com/80/v2-807028a0085530f8271cd8e4cc4c3f74_hd.jpg\",width=400,height=400>  \n",
    "\n",
    "我们假设P(A)=1/3, P(B)=1/3, P(A∩B)=1/6.\n",
    "\n",
    "条件概率P(A|B)就是在B发生的情况下，A发生的概率，即：\n",
    "\n",
    "$$\n",
    "P(A|B)=\\frac{P(A\\cap{B})}{P(B)}=\\frac{1/6}{1/3}=1/2\n",
    "$$\n",
    "\n",
    "\n",
    "也就是说，在B发生时，A有1/2的可能性会发生，我们看上面的图，的确A∩B发生的可能性占到了B全部可能性的一半。\n",
    "\n",
    "将上面的公式与贝叶斯公式进行对比，我们发现：\n",
    "\n",
    "$$\n",
    "\\begin{aligned} % requires amsmath; align* for no eq. number\n",
    "P(A\\cap{B}) & =P(A)P(B|A) \\\\\n",
    "   & = \\frac{1}{3}*\\frac{1}{2}\\\\\n",
    "   & = \\frac{1}{6}\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "而条件概率和贝叶斯公式其实就是上述公式的转换。  \n",
    "\n",
    "## 使用条件概率来分类\n",
    "\n",
    "上面我们提到贝叶斯决策理论要求计算两个概率 $p1(x, y)$ 和 $p2(x, y)$:\n",
    "\n",
    "* 如果$ p1(x, y) > p2(x, y)$, 那么属于类别 1;   \n",
    "* 如果 $p2(x, y) > p1(x, y)$, 那么属于类别 2。     \n",
    "\n",
    "这并不是贝叶斯决策理论的所有内容。使用$ p1() $和 $p2() $只是为了尽可能简化描述，而真正需要计算和比较的是 $p(c_1|x, y) $和 $p(c_2|x, y)$ .这些符号所代表的具体意义是: 给定某个由$x$、$y$ 表示的数据点，那么该数据点来自类别$ c_1 $的概率是多少？数据点来自类别$c_2$ 的概率又是多少？注意这些概率与概率 $p(x, y|c_1)$ 并不一样，不过可以使用贝叶斯准则来交换概率中条件与结果。具体地，应用贝叶斯准则得到:\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "使用上面这些定义，可以定义贝叶斯分类准则为:\n",
    "\n",
    "* 如果 $P(c_1|x, y) > P(c_2|x, y)$, 那么属于类别$ c_1$;   \n",
    "* 如果 $P(c_2|x, y) > P(c_1|x, y)$, 那么属于类别 $c_2$。   \n",
    "\n",
    "在文档分类中，整个文档（如一封电子邮件）是实例，而电子邮件中的某些元素则构成特征。我们可以观察文档中出现的词，并把每个词作为一个特征，而每个词的出现或者不出现作为该特征的值，这样得到的特征数目就会跟词汇表中的词的数目一样多。\n",
    "\n",
    "我们假设特征之间 **相互独立** 。所谓 **独立(independence)** 指的是统计意义上的独立，即一个特征或者单词出现的可能性与它和其他单词相邻没有关系，比如说，“我们”中的“我”和“们”出现的概率与这两个字相邻没有任何关系。这个假设正是朴素贝叶斯分类器中 **朴素(naive)** 一词的含义。朴素贝叶斯分类器中的另一个假设是，每个特征同等重要。\n",
    "\n",
    "Note: 朴素贝叶斯分类器通常有两种实现方式: 一种基于伯努利模型实现，一种基于多项式模型实现。这里采用前一种实现方式。该实现方式中并不考虑词在文档中出现的次数，只考虑出不出现，因此在这个意义上相当于假设词是等权重的。\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "**朴素贝叶斯 开发流程**\n",
    ">收集数据: 可以使用任何方法。  \n",
    "准备数据: 需要数值型或者布尔型数据。   \n",
    "分析数据: 有大量特征时，绘制特征作用不大，此时使用直方图效果更好。  \n",
    "训练算法: 计算不同的独立特征的条件概率。    \n",
    "测试算法: 计算错误率。   \n",
    "使用算法: 一个常见的朴素贝叶斯应用是文档分类。可以在任意的分类场景中使用朴素贝叶斯分类器，不一定   \n",
    "\n",
    "\n",
    "## 使用Python进行文本分类  \n",
    "\n",
    "### 准备数据：从文本中构建词向量\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from numpy import *#词表到向量的转换函数\n",
    "#创建实验样本\n",
    "def loadDataSet():#词条切分后得文档集合\n",
    "    postingList=[['my', 'dog', 'has', 'flea', 'problems', 'help', 'please'],\n",
    "                 ['maybe', 'not', 'take', 'him', 'to', 'dog', 'park', 'stupid'],\n",
    "                 ['my', 'dalmation', 'is', 'so', 'cute', 'I', 'love', 'him'],\n",
    "                 ['stop', 'posting', 'stupid', 'worthless', 'garbage'],\n",
    "                 ['mr', 'licks', 'ate','my', 'steak', 'how', 'to', 'stop', 'him'],\n",
    "                 ['quit', 'buying', 'worthless', 'dog', 'food', 'stupid']]\n",
    "    classVec = [0,1,0,1,0,1]    #1 is abusive, 0 not 类别标签的集合\n",
    "    return postingList,classVec\n",
    "\n",
    "def createVocabList(dataSet):\n",
    "    vocabSet = set([])            #create empty set\n",
    "    for document in dataSet:      #创建两个集合的并集，将文档的新词添加到集合中\n",
    "        vocabSet = vocabSet | set(document) #创建两个集合的并集，将文档中的新词添加到集合中\n",
    "    return list(vocabSet)#返回列表，内容是文档中出现的不重复词的列表\n",
    "\n",
    "def setOfWords2Vec(vocabList, inputSet): # vocabList是已知词汇表，inputSet是某文档\n",
    "    returnVec = [0]*len(vocabList)  # 创建一个列表，内容为0，长度与词汇表等长\n",
    "    for word in inputSet:\n",
    "        if word in vocabList:    # 如果词汇表中的单词在文档中出现过，则将returnVec中对应位置置1\n",
    "            returnVec[vocabList.index(word)] = 1\n",
    "        else: print(\"the word: %s is not in my Vocabulary!\" % word)\n",
    "    return returnVec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([['my', 'dog', 'has', 'flea', 'problems', 'help', 'please'],\n",
       "  ['maybe', 'not', 'take', 'him', 'to', 'dog', 'park', 'stupid'],\n",
       "  ['my', 'dalmation', 'is', 'so', 'cute', 'I', 'love', 'him'],\n",
       "  ['stop', 'posting', 'stupid', 'worthless', 'garbage'],\n",
       "  ['mr', 'licks', 'ate', 'my', 'steak', 'how', 'to', 'stop', 'him'],\n",
       "  ['quit', 'buying', 'worthless', 'dog', 'food', 'stupid']],\n",
       " ['my', 'dog', 'has', 'flea', 'problems', 'help', 'please'])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "listposts,listclass=loadDataSet()\n",
    "listposts,listposts[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['has',\n",
       " 'to',\n",
       " 'posting',\n",
       " 'my',\n",
       " 'stop',\n",
       " 'maybe',\n",
       " 'flea',\n",
       " 'garbage',\n",
       " 'dalmation',\n",
       " 'mr',\n",
       " 'licks',\n",
       " 'ate',\n",
       " 'help',\n",
       " 'how',\n",
       " 'problems',\n",
       " 'food',\n",
       " 'cute',\n",
       " 'love',\n",
       " 'worthless',\n",
       " 'quit',\n",
       " 'buying',\n",
       " 'not',\n",
       " 'take',\n",
       " 'please',\n",
       " 'so',\n",
       " 'him',\n",
       " 'I',\n",
       " 'is',\n",
       " 'steak',\n",
       " 'dog',\n",
       " 'park',\n",
       " 'stupid']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "myvocablist=createVocabList(listposts)\n",
    "myvocablist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "setOfWords2Vec(myvocablist,listposts[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练算法：从词向量计算频率\n",
    "\n",
    "现在已经知道了一个词是否出现在一篇文档中，也知道该文档所属的类别。接下来我们重写贝叶斯准则，将之前的 $x$, $y$ 替换为 $w$. 粗体的$ w$ 表示这是一个向量，即它由多个值组成。在这个例子中，数值个数与词汇表中的词个数相同。\n",
    "\n",
    "$$\n",
    "p(c_i|w)=\\frac{p(w|c_i)p(c_i)}{p(w)}\n",
    "$$\n",
    "\n",
    "\n",
    "我们使用上述公式，对每个类计算该值，然后比较这两个概率值的大小。\n",
    "\n",
    "首先可以通过类别 $i $(侮辱性留言或者非侮辱性留言)中的文档数除以总的文档数来计算概率 $p(c_i)$ 。接下来计算$ p(w | c_i)$ ，这里就要用到朴素贝叶斯假设。如果将 w 展开为一个个独立特征，那么就可以将上述概率写作$ p(w_0, w_1, w_2..._wn | c_i)$ 。这里假设所有词都互相独立，该假设也称作条件独立性假设（例如 $A $和 $B$ 两个人抛骰子，概率是互不影响的，也就是相互独立的，$A$ 抛 2点的同时 $B $抛 3 点的概率就是 1/6 * 1/6），它意味着可以使用 $p(w_0 | c_i)p(w_1 | c_i)p(w_2 | c_i)...p(w_n | c_i)$ 来计算上述概率，这样就极大地简化了计算的过程。\n",
    "\n",
    "伪代码：\n",
    "```\n",
    "计算每个类别中的文档数目\n",
    "对每篇训练文档: \n",
    "    对每个类别: \n",
    "        如果词条出现在文档中-->增加该词条的计数值（for循环或者矩阵相加）\n",
    "        增加所有词条的计数值（此类别下词条总数）\n",
    "对每个类别: \n",
    "    对每个词条: \n",
    "        将该词条的数目除以总词条数目得到的条件概率（P(词条|类别)）\n",
    "返回该文档属于每个类别的条件概率（P(类别|文档的所有词条)）\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#朴素贝叶斯分类器训练函数\n",
    "#trainMatrix为文档矩阵，trainCategory为类别标签向量\n",
    "\n",
    "def trainNB0(trainMatrix,trainCategory):\n",
    "    numTrainDocs = len(trainMatrix)   #得到文档个数\n",
    "    numWords = len(trainMatrix[0])   # 得到特征单词数目\n",
    "    pAbusive = sum(trainCategory)/float(numTrainDocs)  # 侮辱性文件的出现概率，即trainCategory中所有的1的个数，\n",
    "                                                       # 代表的就是多少个侮辱性文件，与文件的总数相除就得到了侮辱性文件的出现概率\n",
    "    p0Num = ones(numWords); p1Num = ones(numWords)      #构造单词出现的次数列表 \n",
    "    p0Denom = 2.0; p1Denom = 2.0                        #整个数据集单词出现总数\n",
    "                                                        # 整个数据集单词出现总数，2.0根据样本/实际调查结果调整分母的值（2主要是避免分母为0，当然值可以调整）\n",
    "                                                        # p0Denom 正常的统计\n",
    "                                                        # p1Denom 侮辱的统计\n",
    "    for i in range(numTrainDocs):\n",
    "        if trainCategory[i] == 1:\n",
    "            p1Num += trainMatrix[i]  # 如果是侮辱性文件，对侮辱性文件的向量进行加和\n",
    "            p1Denom += sum(trainMatrix[i])  # 对向量中的所有元素进行求和，也就是计算所有侮辱性文件中出现的单词总数\n",
    "        else:\n",
    "            p0Num += trainMatrix[i]\n",
    "            p0Denom += sum(trainMatrix[i])\n",
    "    p1Vect = log(p1Num/p1Denom)          # 类别1，即侮辱性文档的[P(F1|C1),P(F2|C1),P(F3|C1),P(F4|C1),P(F5|C1)....]列表\n",
    "                                         # 即 在1类别下，每个单词出现的概率\n",
    "    p0Vect = log(p0Num/p0Denom)          # 类别0，即正常文档的[P(F1|C0),P(F2|C0),P(F3|C0),P(F4|C0),P(F5|C0)....]列表\n",
    "                                         # 即 在0类别下，每个单词出现的概率\n",
    "    return p0Vect,p1Vect,pAbusive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  0],\n",
       " [0,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  1,\n",
       "  1],\n",
       " [0,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0],\n",
       " [0,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  1],\n",
       " [0,\n",
       "  1,\n",
       "  0,\n",
       "  1,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  0],\n",
       " [0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  1,\n",
       "  0,\n",
       "  1]]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainMat=[]\n",
    "for postinDoc in listposts:\n",
    "    trainMat.append(setOfWords2Vec(myvocablist,postinDoc))\n",
    "trainMat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "p0v,p1v,pab=trainNB0(trainMat,listclass)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-2.56494936, -2.56494936, -3.25809654, -1.87180218, -2.56494936,\n",
       "       -3.25809654, -2.56494936, -3.25809654, -2.56494936, -2.56494936,\n",
       "       -2.56494936, -2.56494936, -2.56494936, -2.56494936, -2.56494936,\n",
       "       -3.25809654, -2.56494936, -2.56494936, -3.25809654, -3.25809654,\n",
       "       -3.25809654, -3.25809654, -3.25809654, -2.56494936, -2.56494936,\n",
       "       -2.15948425, -2.56494936, -2.56494936, -2.56494936, -2.56494936,\n",
       "       -3.25809654, -3.25809654])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p0v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-3.04452244, -2.35137526, -2.35137526, -3.04452244, -2.35137526,\n",
       "       -2.35137526, -3.04452244, -2.35137526, -3.04452244, -3.04452244,\n",
       "       -3.04452244, -3.04452244, -3.04452244, -3.04452244, -3.04452244,\n",
       "       -2.35137526, -3.04452244, -3.04452244, -1.94591015, -2.35137526,\n",
       "       -2.35137526, -2.35137526, -2.35137526, -3.04452244, -3.04452244,\n",
       "       -2.35137526, -3.04452244, -3.04452244, -3.04452244, -1.94591015,\n",
       "       -2.35137526, -1.65822808])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p1v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "def classifyNB(vec2Classify, p0Vec, p1Vec, pClass1):\n",
    "    \"\"\"\n",
    "    使用算法：\n",
    "        # 将乘法转换为加法\n",
    "        乘法：P(C|F1F2...Fn) = P(F1F2...Fn|C)P(C)/P(F1F2...Fn)\n",
    "        加法：P(F1|C)*P(F2|C)....P(Fn|C)P(C) -> log(P(F1|C))+log(P(F2|C))+....+log(P(Fn|C))+log(P(C))\n",
    "    :param vec2Classify: 待测数据[0,1,1,1,1...]，即要分类的向量\n",
    "    :param p0Vec: 类别0，即正常文档的[log(P(F1|C0)),log(P(F2|C0)),log(P(F3|C0)),log(P(F4|C0)),log(P(F5|C0))....]列表\n",
    "    :param p1Vec: 类别1，即侮辱性文档的[log(P(F1|C1)),log(P(F2|C1)),log(P(F3|C1)),log(P(F4|C1)),log(P(F5|C1))....]列表\n",
    "    :param pClass1: 类别1，侮辱性文件的出现概率\n",
    "    :return: 类别1 or 0\n",
    "    \"\"\"\n",
    "    # 计算公式  log(P(F1|C))+log(P(F2|C))+....+log(P(Fn|C))+log(P(C))\n",
    "    # 大家可能会发现，上面的计算公式，没有除以贝叶斯准则的公式的分母，也就是 P(w) （P(w) 指的是此文档在所有的文档中出现的概率）就进行概率大小的比较了，\n",
    "    # 因为 P(w) 针对的是包含侮辱和非侮辱的全部文档，所以 P(w) 是相同的。\n",
    "    # 使用 NumPy 数组来计算两个向量相乘的结果，这里的相乘是指对应元素相乘，即先将两个向量中的第一个元素相乘，然后将第2个元素相乘，以此类推。\n",
    "    # 我的理解是：这里的 vec2Classify * p1Vec 的意思就是将每个词与其对应的概率相关联起来\n",
    "    p1 = sum(vec2Classify * p1Vec) + log(pClass1)    # P(w|c1) * P(c1) ，即贝叶斯准则的分子\n",
    "    p0 = sum(vec2Classify * p0Vec) + log(1.0 - pClass1)  # P(w|c0) * P(c0) ，即贝叶斯准则的分子·\n",
    "    if p1 > p0:\n",
    "        return 1\n",
    "    else: \n",
    "        return 0\n",
    "\n",
    "def testingNB():\n",
    "    \"\"\"\n",
    "    测试朴素贝叶斯算法\n",
    "    \"\"\"\n",
    "    # 1. 加载数据集\n",
    "    listOPosts,listClasses = loadDataSet()\n",
    "    # 2. 创建单词集合\n",
    "    myVocabList = createVocabList(listOPosts)\n",
    "    # 3. 计算单词是否出现并创建数据矩阵\n",
    "    trainMat=[]\n",
    "    for postinDoc in listOPosts:\n",
    "        # 返回m*len(myVocabList)的矩阵， 记录的都是0，1信息\n",
    "        trainMat.append(setOfWords2Vec(myVocabList, postinDoc))\n",
    "    # 4. 训练数据\n",
    "    p0V,p1V,pAb = trainNB0(array(trainMat),array(listClasses))\n",
    "    # 5. 测试数据\n",
    "    testEntry = ['love', 'my', 'dalmation']\n",
    "    thisDoc = array(setOfWords2Vec(myVocabList, testEntry))\n",
    "    print(testEntry,'classified as: ',classifyNB(thisDoc,p0V,p1V,pAb))\n",
    "    testEntry = ['stupid', 'garbage']\n",
    "    thisDoc = array(setOfWords2Vec(myVocabList, testEntry))\n",
    "    print(testEntry,'classified as: ',classifyNB(thisDoc,p0V,p1V,pAb))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['love', 'my', 'dalmation'] classified as:  0\n",
      "['stupid', 'garbage'] classified as:  1\n"
     ]
    }
   ],
   "source": [
    "testingNB()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "        \n",
    "def bagOfWords2VecMN(vocabList, inputSet):\n",
    "    returnVec = [0]*len(vocabList)\n",
    "    for word in inputSet:\n",
    "        if word in vocabList:\n",
    "            returnVec[vocabList.index(word)] += 1\n",
    "    return returnVec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def textParse(bigString):    #input is big string, #output is word list\n",
    "    import re\n",
    "    listOfTokens = re.split(r'\\W*', bigString)\n",
    "    return [tok.lower() for tok in listOfTokens if len(tok) > 2] \n",
    "    \n",
    "def spamTest():\n",
    "    docList=[]; classList = []; fullText =[]\n",
    "    for i in range(1,26):\n",
    "        wordList = textParse(open('email/spam/%d.txt' % i).read())\n",
    "        docList.append(wordList)\n",
    "        fullText.extend(wordList)\n",
    "        classList.append(1)\n",
    "        wordList = textParse(open('email/ham/%d.txt' % i).read())\n",
    "        docList.append(wordList)\n",
    "        fullText.extend(wordList)\n",
    "        classList.append(0)\n",
    "    vocabList = createVocabList(docList)#create vocabulary\n",
    "    trainingSet = range(50); testSet=[]           #create test set\n",
    "    for i in range(10):\n",
    "        randIndex = int(random.uniform(0,len(trainingSet)))\n",
    "        testSet.append(trainingSet[randIndex])\n",
    "        del(trainingSet[randIndex])  \n",
    "    trainMat=[]; trainClasses = []\n",
    "    for docIndex in trainingSet:#train the classifier (get probs) trainNB0\n",
    "        trainMat.append(bagOfWords2VecMN(vocabList, docList[docIndex]))\n",
    "        trainClasses.append(classList[docIndex])\n",
    "    p0V,p1V,pSpam = trainNB0(array(trainMat),array(trainClasses))\n",
    "    errorCount = 0\n",
    "    for docIndex in testSet:        #classify the remaining items\n",
    "        wordVector = bagOfWords2VecMN(vocabList, docList[docIndex])\n",
    "        if classifyNB(array(wordVector),p0V,p1V,pSpam) != classList[docIndex]:\n",
    "            errorCount += 1\n",
    "            print(\"classification error\",docList[docIndex])\n",
    "    print('the error rate is: ',float(errorCount)/len(testSet))\n",
    "    #return vocabList,fullText"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "\n",
    "### 朴素贝叶斯分类\n",
    "那我们怎么来利用贝叶斯定理来进行分类呢？\n",
    "\n",
    "我们首先假设有N种可能的分类类别，分别为${c1,c2,c3...cN}$，这时候对于样本$x$，我们利用贝叶斯定理：  \n",
    "\n",
    "$$\n",
    "P(c|x)=\\frac{P(c)P(x|c)}{P{x}}\n",
    "$$  \n",
    "\n",
    "我们需要找到最大的$P(c|x)$作为样本x的最终分类，即：\n",
    "\n",
    "$$\n",
    "P(c|x)=argmax\\frac{P(c)P(x|c)}{P(x)}\n",
    "$$\n",
    "\n",
    "其中$P(c)$表示训练样本空间中各类样本所占的比例；$P(x|c)$表示类别为$c$的条件下，样本$x$的类条件概率；$P(x)$是用来归一化的“证据因子”，对于给定样本$x$，证据因子$P(x)$与类标记无关。\n",
    "\n",
    "因此，估计$P(c|x)$的问题就转化为如何基于训练数据D来估计$P(c)$和$P(x|c)$。\n",
    "\n",
    "$P(c)$很容易就可以得到，令$D_c$表示训练集$D$中第$c$类样本组成的集合，若有充足的独立同分布样本，则我们可以很容易估计出$P(c)$\n",
    "\n",
    "\n",
    "$$\n",
    "P(c)=\\frac{|D_c|}{|D|}\n",
    "$$\n",
    "不难发现，基于贝叶斯公式，主要是$P(x|c)$比较难以求解，我们一般估计类条件概率主要使用极大似然估计法，但因为这里是所有属性上的联合概率，难以运用最大似然估计法直接估计得到。为了避开这个障碍，朴素贝叶斯分类器运用了“属性条件独立性假设”：对已知类别，假设所有属性相互独立。\n",
    "\n",
    "基于上述假设，贝叶斯分类公式可以重写为：\n",
    "\n",
    "$$\n",
    "P(c|x)=\\frac{P(c)P(x|c)}{P(x)}=\\frac{P(c)}{P(x)}\\prod_{i=1}^{d}P(x_i|c)\n",
    "$$\n",
    "\n",
    "其中$d$为属性的数目，$x_i$为样本$x$在第$i$个属性上的取值。\n",
    "\n",
    "由于对所有的类别来讲，$P(x$)都相同，所以基于上式的贝叶斯判定准则为：\n",
    "\n",
    "\n",
    "$$\n",
    "h_{nb}=argmax_{c\\in{c_1,c_2,\\dots,c_N}}P(c)\\prod_{i=1}^{d}P(x_i|c)\n",
    "$$\n",
    "\n",
    "\n",
    "对于离散属性而言，令$D_{c,x_i}$表示在第$i$个属性上取值为$x_i$的样本组成的集合，则类条件概率可以估计为：\n",
    "\n",
    "\n",
    "$$\n",
    "P(x_i|c)=\\frac{|D_{c,x_i}|}{|D_c|}\n",
    "$$\n",
    "\n",
    "\n",
    "举例如下：  \n",
    "\n",
    "|是否会飞|是否会走|是否会跑|是不是鸟|\n",
    "|--|--|--|--|\n",
    "|1|1|1|是|\n",
    "|1|0|0|是|\n",
    "|0|1|1|是|\n",
    "|0|1|0|否|\n",
    "|1|0|1|否|\n",
    "\n",
    "\n",
    "上面是训练样本，对于给定$x$样本=${1,1,0}$，请问它是不是鸟？\n",
    "\n",
    "首先我们估计先验概率$P(c)$，有：\n",
    "\n",
    "$$P(是鸟=是)=3/5$$\n",
    "$$P(是鸟=否)=2/5$$\n",
    "\n",
    "这时候我们为每个属性计算条件概率$P(xi|c)$ \n",
    "\n",
    "<img src=\"https://pic2.zhimg.com/80/v2-b2efb3940bc8f971070759241198f428_hd.jpg\">\n",
    "\n",
    "\n",
    "于是，有   \n",
    "\n",
    "\n",
    "<img src=\"https://pic3.zhimg.com/80/v2-4e45e65f8fa28215495d2bbf3946157a_hd.jpg\",width=500,height=500>\n",
    "\n",
    "\n",
    "由于$0.177>0.05$,因此朴素贝叶斯分类将其判定为“是鸟”。\n",
    "\n",
    "  \n",
    "**平滑处理**  \n",
    "\n",
    "我们注意到，上面计算过程存在一个弊端，那就是，如果某个属性值在训练集中没有与某个类同时出现，则连乘时计算出的概率值为零。  \n",
    "\n",
    "为了让大家理解，还是用上面的例子，把表格稍微修改下，如下：  \n",
    "\n",
    "|是否会飞|是否会走|是否会跑|是不是鸟|\n",
    "|--|--|--|--|\n",
    "|1|1|1|是|\n",
    "|1|0|0|是|\n",
    "|0|1|1|是|\n",
    "|0|0|0|否|\n",
    "|1|0|1|否|\n",
    "\n",
    "只对第四组数据做了一点改变，这时候我们计算测试例，有：   \n",
    "\n",
    "$$\n",
    "P(会走=1|是鸟=否)=0\n",
    "$$\n",
    "\n",
    "因此，无论该样本的其他属性是什么，“不是鸟”的概率都为零，分类结果都将是“是鸟=是”，这显然不合理。  \n",
    "\n",
    "为了避免上述弊端的出现，我们在计算概率值时要进行平滑处理，常用“拉普拉斯修正”：\n",
    "\n",
    "$$\n",
    "\\hat{P}(c)=\\frac{|D_c|+1}{|D|+N}\n",
    "$$\n",
    "\n",
    "\n",
    "$$\n",
    "\\hat{P}(x_i|c)=\\frac{|D_{c,x_i}|+1}{|D_c|+N_i}\n",
    "$$\n",
    "\n",
    "\n",
    "其中，$N$表示训练集$D$中可能出现的类别数，$N_i$表示第$i$个属性可能的取值数。拉普拉斯修正避免了因训练集样本不充分而导致的概率值为零的问题。\n",
    "\n",
    "\n",
    "## 半朴素贝叶斯分类   \n",
    "\n",
    "我们在朴素贝叶斯分类中采用了“属性条件独立性假设”，但是在现实中，这个假设往往很难成立。\n",
    "\n",
    "所以，半朴素贝叶斯分类的基本思想是适当考虑一部分属性间的依赖关系。\n",
    "\n",
    "“独依赖估计”是半朴素贝叶斯分类中常用的一种策略。所谓的“独依赖”就是假设每个属性在类别之外最多依赖与其中一个属性，即  \n",
    "\n",
    "$$\n",
    "P(c|x)正比于P(c)\\prod_{i=1}^{d}P(x_i|c,p\\alpha_i)\n",
    "$$\n",
    "\n",
    "其中，pai是属性xi所依赖的属性，称为xi的父属性。\n",
    "\n",
    "怎么来确定属性间的关系呢？一般有三种方法，分别是：SPODE、TAN、AODE.\n",
    "\n",
    "我们这里主要讲一下TAN方法，另外两种方法想要了解的同学可以参照周志华老师《机器学习》P155.\n",
    "\n",
    "TAN方法中最重要的一个判断标准就是两个属性间的条件互信息：\n",
    "\n",
    "$$\n",
    "I(x_i,x_j|y)=\\sum_{x_i,x_j\\in{y}}P(x_i,x_j|c)log\\frac{P(x_i,x_j|c)}{P(x_i|c)P(x_j|c)}\n",
    "$$\n",
    "\n",
    "条件互信息刻画了属性xi和xj在已知类别下的相关性。我们需要找的，就是相关性大的属性。\n",
    "\n",
    "\n",
    "\n",
    "**EM算法**\n",
    "\n",
    "我们在现实中常常会遇到“不完整”的训练样本，在这种存在“未观测”变量的情形下，我们如何对模型参数进行估计？即如何用极大似然估计法计算模型参数。本算法与贝叶斯分类算法关系不大，这里我们仅对EM算法作简单介绍，想要详细了解的同学可以自行查阅资料。\n",
    "\n",
    "我们定义未观测变量的学名是“隐变量”。令X表示已观测变量集，Z表示隐变量集，theta表示模型参数。\n",
    "\n",
    "EM算法(参数最大化算法)是常用的估计参数隐变量的利器\n",
    "\n",
    "其主要思想只有两步：\n",
    "\n",
    "1. 第一步是期望(E)步，利用当前估计的参数值来计算对数似然的期望值\n",
    "\n",
    "2. 第二步是最大化(M)步，寻找能使E步产生的似然期望最大化的参数值。然后，新得到的参数值重新被用于E步。\n",
    "\n",
    "重复上述两步，直到收敛到局部最优解。就可以得到模型的参数。\n",
    "\n",
    "\n",
    "\n",
    "上面就是贝叶斯分类的全部原理部分。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}