part1/pgmpy/ch01/1_Bayesian_Network_Fundamentals.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 1: Bayesian Network Fundamentals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 싸이그래머 / QGM - pgmpy [1]\n",
    "* 김무성"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Probability theory\n",
    "* Installing tools \n",
    "* Representing independencies using pgmpy \n",
    "* Representing joint probability distributions using pgmpy \n",
    "* Conditional probability distribution \n",
    "* Graph theory \n",
    "* Bayesian models \n",
    "* Relating graphs and distributions \n",
    "* CPD representations\n",
    "* Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Probability theory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Random variable \n",
    "* Independence and conditional independence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 참고\n",
    "* [4] stanford-pgm/slides/1.1.2-Intro-distributions - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/1.1.2-Intro-distributions.pdf\n",
    "* [5] stanford-pgm/slides/1.1.3-Intro-factors - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/1.1.3-Intro-factors.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random variable "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A random variable is a way of representing an attribute of the outcome."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Formally, a random variable X is a function that maps a possible set of outcomes Ω to some set E, which is represented as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "X :Ω →E"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random variables can either be discrete or continuous."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For any event whose outcome is represented by some random variable (X), we can assign some value to each of the possible outcomes of X, which represents how probable it is."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is known as the probability distribution of the random variable and is denoted by P(X)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Independence and conditional independence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 참고\n",
    "* [6] stanford-pgm/slides/2.1.4-Repn-Ind-conditional-independence - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/2.1.4-Repn-Ind-conditional-independence.pdf\n",
    "* [8] stanford-pgm/slides/2.1.2-Repn-BNs-patterns - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/2.1.2-Repn-BNs-patterns.pdf\n",
    "* [9] stanford-pgm/slides/2.1.3-Repn-BNs-flow-influence - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/2.1.3-Repn-BNs-flow-influence.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### joint probability distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "P(X1,X2,...,Xn )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Independence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.1.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### conditional independence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Q ⊥ N | C"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Installing tools "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* IPython \n",
    "* pgmpy "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 참고 : Docker Setting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 각자 도커 기반 셋팅이 되어 있어야 함. (리눅스의 경우 docker 시스템 셋팅, 맥과 윈도우의 경우 docker toolbox 셋팅. 이와 관련 문서는 -https://gist.github.com/haje01/0fb6d63bf065c9831256)\n",
    "2. 실습 이미지 다운로드\n",
    "    - <font color=\"blue\">docker pull psygrammer/pgmpy_jupyter</font>\n",
    "3. 컨테이너 실행\n",
    "    - <font color=\"blue\">docker run -d -p 8888:8888 -e GRANT_SUDO=yes --name run_pgmpy psygrammer/pgmpy_jupyter</font>\n",
    "        - (참고) 다음의 문서를 참조한다. 컨테이너 실행 옵션들이 나옴 - https://github.com/jupyter/docker-stacks/tree/master/scipy-notebook\n",
    "        - 컨테이너 실행\n",
    "        - // 슈도유저 권한주고(권장),\n",
    "        - // 별명도 붙여줬음(이건 옵션)\n",
    "        - // docker run -d -p [접속_포트]:8888 -e GRANT_SUDO=yes --name \n",
    "        - // [컨테이너별명] psygrammer/pgmpy_jupyter\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 이 셋팅은 다음(Ipython, pgmpy 항목)설명을 실행해서 만들어진 것임."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IPython  "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">> pip install ipython"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## pgmpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* pgmpy is a Python library to work with Probabilistic Graphical models.\n",
    "* As it's currently not on PyPi, we will need to build it manually."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">> git clone https://github.com/pgmpy/pgmpy"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">> cd pgmpy"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">> git checkout book/v0.1"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ">> sudo python3 setup.py install"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Representing independencies using pgmpy "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* To represent independencies, pgmpy has two classes, namely \n",
    "    - IndependenceAssertion and \n",
    "    - Independencies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### IndependenceAssertion class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Firstly we need to import IndependenceAssertion\n",
    "from pgmpy.independencies import IndependenceAssertion\n",
    "# Each assertion is in the form of [X, Y, Z] \n",
    "# meaning X is independent of Y given Z."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(X _|_ Y)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assertion1 = IndependenceAssertion('X', 'Y')\n",
    "assertion1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(X _|_ Y | Z)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To represent conditional assertions, \n",
    "# we just need to add a third argument \n",
    "# to IndependenceAssertion:\n",
    "assertion2 = IndependenceAssertion('X', 'Y', 'Z')\n",
    "assertion2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(X _|_ Y | A, B)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# IndependenceAssertion also allows us to represent assertions \n",
    "# in the form of (X ⊥Y,Z | A,B). \n",
    "# To do this, we just need to pass \n",
    "# a list of random variables as arguments:\n",
    "assertion3 = IndependenceAssertion('X', 'Y', ['A', 'B'])\n",
    "assertion3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Independencies class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Moving on to the Independencies class, \n",
    "# an Independencies object is used to represent a set of assertions. \n",
    "# Often, in the case of Bayesian or Markov networks,\n",
    "# we have more than one assertion corresponding to a given model, \n",
    "# and to represent these independence assertions for the models, \n",
    "# we generally use the Independencies object. \n",
    "#Let's take a few examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pgmpy.independencies import Independencies\n",
    "# There are multiple ways to create an Independencies object, we\n",
    "# could either initialize an empty object or initialize with some\n",
    "# assertions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "independencies = Independencies() # Empty object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "independencies.get_assertions()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "independencies.add_assertions(assertion1, assertion2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(X _|_ Y), (X _|_ Y | Z)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "independencies.get_assertions()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(X _|_ Y)\n",
       "(X _|_ Y | Z)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can also directly initialize Independencies \n",
    "# in these two ways:\n",
    "independencies = Independencies(assertion1, assertion2)\n",
    "independencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(X _|_ Y)\n",
       "(A _|_ B | C)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "independencies = Independencies(['X', 'Y'],\n",
    "                                ['A', 'B', 'C'])\n",
    "independencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(X _|_ Y), (A _|_ B | C)]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "independencies.get_assertions()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Representing joint probability distributions using pgmpy "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### JointProbabilityDistribution class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We can also represent joint probability distributions \n",
    "# using pgmpy's JointProbabilityDistribution class. \n",
    "# Let's say we want to represent the joint distribution \n",
    "# over the outcomes of tossing two fair coins. \n",
    "# So, in this case, the probability of all the possible outcomes \n",
    "# would be 0.25, which is shown as follows:\n",
    "from pgmpy.factors import JointProbabilityDistribution as Joint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "distribution = Joint(['coin1', 'coin2'],\n",
    "                                 [2, 2],\n",
    "                                 [0.25, 0.25, 0.25, 0.25])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "╒═════════╤═════════╤══════════════════╕\n",
      "│ coin1   │ coin2   │   P(coin1,coin2) │\n",
      "╞═════════╪═════════╪══════════════════╡\n",
      "│ coin1_0 │ coin2_0 │           0.2500 │\n",
      "├─────────┼─────────┼──────────────────┤\n",
      "│ coin1_0 │ coin2_1 │           0.2500 │\n",
      "├─────────┼─────────┼──────────────────┤\n",
      "│ coin1_1 │ coin2_0 │           0.2500 │\n",
      "├─────────┼─────────┼──────────────────┤\n",
      "│ coin1_1 │ coin2_1 │           0.2500 │\n",
      "╘═════════╧═════════╧══════════════════╛\n"
     ]
    }
   ],
   "source": [
    "print(distribution)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can also conduct independence queries \n",
    "# over these distributions in pgmpy:\n",
    "distribution.check_independence('coin1', 'coin2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conditional probability distribution "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Representing CPDs using pgmpy "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.2.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin by representing the marginal distribution of the quality of food with Q. As we mentioned earlier, it can be categorized into three values {good, bad, average}. For example, P(Q) can be represented in the tabular form as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.3.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, let's say P(L) is the probability distribution of the location of the restaurant. Its CPD can be represented as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.4.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the cost of restaurant C depends on both the quality of food Q and its location L, we will be considering P(C | Q, L), which is the conditional distribution of C, given Q and L:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.5.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Representing CPDs using pgmpy "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Let's first see how to represent the tabular CPD \n",
    "# using pgmpy for variables that have no conditional variables:\n",
    "\n",
    "from pgmpy.factors import TabularCPD\n",
    "# For creating a TabularCPD object we need to pass three\n",
    "# arguments: the variable name, its cardinality that is the number\n",
    "# of states of the random variable and the probability value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# corresponding each state.\n",
    "quality = TabularCPD(variable='Quality',\n",
    "                    variable_card=3,\n",
    "                    values=[[0.3], [0.5], [0.2]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "╒════════════════╤═════╕\n",
      "│ ['Quality', 0] │ 0.3 │\n",
      "├────────────────┼─────┤\n",
      "│ ['Quality', 1] │ 0.5 │\n",
      "├────────────────┼─────┤\n",
      "│ ['Quality', 2] │ 0.2 │\n",
      "╘════════════════╧═════╛\n"
     ]
    }
   ],
   "source": [
    "print(quality)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('Quality',\n",
       "              [State(var='Quality', state=0),\n",
       "               State(var='Quality', state=1),\n",
       "               State(var='Quality', state=2)])])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quality.variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quality.cardinality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.3,  0.5,  0.2])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quality.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# You can see here that the values of the CPD are a 1D array instead of a 2D array, which you passed as an argument.\n",
    "# Actually, pgmpy internally stores the values of the TabularCPD as a flattened numpy array. \n",
    "# We will see the reason for this in the next chapter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "location = TabularCPD(variable='Location',\n",
    "                                 variable_card=2,\n",
    "                                 values=[[0.6], [0.4]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "╒═════════════════╤═════╕\n",
      "│ ['Location', 0] │ 0.6 │\n",
      "├─────────────────┼─────┤\n",
      "│ ['Location', 1] │ 0.4 │\n",
      "╘═════════════════╧═════╛\n"
     ]
    }
   ],
   "source": [
    "print(location)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# However, when we have conditional variables, \n",
    "# we also need to specify them and the cardinality of those variables. \n",
    "# Let's define the TabularCPD for the cost variable:\n",
    "cost = TabularCPD(\n",
    "                         variable='Cost',\n",
    "                         variable_card=2,\n",
    "                         values=[[0.8, 0.6, 0.1, 0.6, 0.6, 0.05],\n",
    "                                 [0.2, 0.4, 0.9, 0.4, 0.4, 0.95]],\n",
    "                         evidence=['Q', 'L'],\n",
    "                         evidence_card=[3, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "╒═════════════╤════════════╤════════════╤════════════╤════════════╤════════════╤════════════╕\n",
      "│ L           │ ['L', '0'] │ ['L', '0'] │ ['L', '0'] │ ['L', '1'] │ ['L', '1'] │ ['L', '1'] │\n",
      "├─────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤\n",
      "│ Q           │ ['Q', '0'] │ ['Q', '1'] │ ['Q', '2'] │ ['Q', '0'] │ ['Q', '1'] │ ['Q', '2'] │\n",
      "├─────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤\n",
      "│ ['Cost', 0] │ 0.8        │ 0.6        │ 0.1        │ 0.6        │ 0.6        │ 0.05       │\n",
      "├─────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤\n",
      "│ ['Cost', 1] │ 0.2        │ 0.4        │ 0.9        │ 0.4        │ 0.4        │ 0.95       │\n",
      "╘═════════════╧════════════╧════════════╧════════════╧════════════╧════════════╧════════════╛\n"
     ]
    }
   ],
   "source": [
    "print(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Graph theory  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Nodes and edges \n",
    "* Walk, paths, and trails"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "G = (V, E)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nodes and edges "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.6.png\" width=600/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Walk, paths, and trails"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### walk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "W : a, c, d, c, e:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.7.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### cyclic & acyclic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, a graph is known as cyclic if there are one or more paths that start and end at the same node. Such paths are known as cycles. Similarly, if there are no cycles in a graph, it is known as an acyclic graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bayesian models "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Representation \n",
    "* Factorization of a distribution over a network \n",
    "* Implementing Bayesian networks using pgmpy \n",
    "* Reasoning pattern in Bayesian networks \n",
    "* D-separation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 참고\n",
    "* [7] stanford-pgm/slides/2.1.1-Repn-BNs-semantics - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/2.1.1-Repn-BNs-semantics.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.8.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.9.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Representation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* A Bayesian network is represented by a \n",
    "    - Directed Acyclic Graph (DAG) and \n",
    "    - a set of Conditional Probability Distributions (CPD) in which:\n",
    "        - The nodes represent random variables\n",
    "        - The edges represent dependencies\n",
    "        - For each of the nodes, we have a CPD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* In our previous restaurant example, the nodes would be as follows:\n",
    "    - Quality of food (Q)\n",
    "    - Location (L)\n",
    "    - Cost of food (C)\n",
    "    - Number of people (N)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.9.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Factorization of a distribution over a network "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.10.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.11.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now, looking into the Bayesian network (BN) for the restaurant, we can say that for any Bayesian network, the joint probability distribution P(X1, X2 ,..., Xn ) over all its random variables {X1, X2 ,..., Xn} can be represented as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### chain rule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.12.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### factorization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, we say that a distribution P factorizes over a graph G, if P can be encoded as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.13.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementing Bayesian networks using pgmpy "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Bayesian model representation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.14.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bayesian model representation "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# In pgmpy, we can initialize an empty BN or a model with nodes and edges. \n",
    "# We can initializing an empty model as follows:\n",
    "from pgmpy.models import BayesianModel\n",
    "model = BayesianModel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We can now add nodes and edges to this network:\n",
    "model.add_nodes_from(['rain', 'traffic_jam'])\n",
    "model.add_edge('rain', 'traffic_jam')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# If we add an edge, but the nodes, between which the edge is, \n",
    "# are not present in the model, \n",
    "# pgmpy automatically adds those nodes to the model.\n",
    "model.add_edge('accident', 'traffic_jam')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['traffic_jam', 'rain', 'accident']"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.nodes()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('rain', 'traffic_jam'), ('accident', 'traffic_jam')]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.edges()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# In the case of a Bayesian network, each of the nodes has an associated CPD with it. \n",
    "# So, let's define some tabular CPDs to associate with the model:\n",
    "from pgmpy.factors import TabularCPD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cpd_rain = TabularCPD('rain', 2, [[0.4], [0.6]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cpd_accident = TabularCPD('accident', 2, [[0.2], [0.8]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cpd_traffic_jam = TabularCPD(\n",
    "                            'traffic_jam', 2,\n",
    "                            [[0.9, 0.6, 0.7, 0.1],\n",
    "                             [0.1, 0.4, 0.3, 0.9]],\n",
    "                            evidence=['rain', 'accident'],\n",
    "                            evidence_card=[2, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Here, we defined three CPDs. We now need to associate them with our model. \n",
    "# To associate them with the model, we just need to use the add_cpd method \n",
    "# and pgmpy automatically figures out which CPD is for which node:\n",
    "model.add_cpds(cpd_rain, cpd_accident, cpd_traffic_jam)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<TabularCPD representing P(rain:2) at 0x7fbb63fd0908>,\n",
       " <TabularCPD representing P(accident:2) at 0x7fbb63fd0cf8>,\n",
       " <TabularCPD representing P(traffic_jam:2 | rain:2, accident:2) at 0x7fbb63fd0c50>]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.get_cpds()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Now, let's add the remaining variables and their CPDs:\n",
    "model.add_node('long_queues')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.add_edge('traffic_jam', 'long_queues')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cpd_long_queues = TabularCPD('long_queues', 2,\n",
    "                                         [[0.9, 0.2],\n",
    "                                          [0.1, 0.8]],\n",
    "                                         evidence=['traffic_jam'],\n",
    "                                         evidence_card=[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.add_cpds(cpd_long_queues)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.add_nodes_from(['getting_up_late',\n",
    "                                  'late_for_school'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.add_edges_from(\n",
    "                      [('getting_up_late', 'late_for_school'),\n",
    "                       ('traffic_jam', 'late_for_school')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cpd_getting_up_late = TabularCPD('getting_up_late', 2,\n",
    "                                             [[0.6], [0.4]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cpd_late_for_school = TabularCPD(\n",
    "                                  'late_for_school', 2,\n",
    "                                  [[0.9, 0.45, 0.8, 0.1],\n",
    "                                   [0.1, 0.55, 0.2, 0.9]],\n",
    "                                  evidence=['getting_up_late',\n",
    "                                            'traffic_jam'],\n",
    "                                  evidence_card=[2, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.add_cpds(cpd_getting_up_late, cpd_late_for_school)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<TabularCPD representing P(rain:2) at 0x7fbb63fd0908>,\n",
       " <TabularCPD representing P(accident:2) at 0x7fbb63fd0cf8>,\n",
       " <TabularCPD representing P(traffic_jam:2 | rain:2, accident:2) at 0x7fbb63fd0c50>,\n",
       " <TabularCPD representing P(long_queues:2 | traffic_jam:2) at 0x7fbb63fd0ba8>,\n",
       " <TabularCPD representing P(getting_up_late:2) at 0x7fbb63fd0ac8>,\n",
       " <TabularCPD representing P(late_for_school:2 | getting_up_late:2, traffic_jam:2) at 0x7fbb63fd0ef0>]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.get_cpds()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Additionally, pgmpy also provides a check_model method \n",
    "# that checks whether the model \n",
    "# and all the associated CPDs are consistent:\n",
    "model.check_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# In case we have got some wrong CPD associated with the model \n",
    "# and we want to remove it, we can use the remove_cpd method. \n",
    "# Let's say we want to remove the CPD associated with variable late_for_school, \n",
    "# we could simply do as follows:\n",
    "model.remove_cpds('late_for_school')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<TabularCPD representing P(rain:2) at 0x7fbb63fd0908>,\n",
       " <TabularCPD representing P(accident:2) at 0x7fbb63fd0cf8>,\n",
       " <TabularCPD representing P(traffic_jam:2 | rain:2, accident:2) at 0x7fbb63fd0c50>,\n",
       " <TabularCPD representing P(long_queues:2 | traffic_jam:2) at 0x7fbb63fd0ba8>,\n",
       " <TabularCPD representing P(getting_up_late:2) at 0x7fbb63fd0ac8>]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.get_cpds()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reasoning pattern in Bayesian networks "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 참고\n",
    "* [8] stanford-pgm/slides/2.1.2-Repn-BNs-patterns - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/2.1.2-Repn-BNs-patterns.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the chain rule of the Bayesian network, we have the joint probability\n",
    "distribution P as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.15.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.14.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### causal reasoning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "P(J|A = True)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### evidential reasoning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "P(R|J = True) ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### intercausal reasoning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "R = False and J = True."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "P(A = True|J = True, R = False) ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## D-separation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Direct connection \n",
    "* Indirect connection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will discuss the independence conditions that hold in a Bayesian network no matter which probability distribution is parameterizing the network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 참고\n",
    "* [9] stanford-pgm/slides/2.1.3-Repn-BNs-flow-influence - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/2.1.3-Repn-BNs-flow-influence.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* bayes ball"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://feature-space.com/en/image44.a73eb.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Direct connection "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "X -> Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indirect connection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Indirect causal effect\n",
    "* Indirect evidential effect\n",
    "* Common cause\n",
    "* Common effect"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.16.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### active trail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, if an influence can flow in a trail in a network, it is known as an active trail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see some examples to check the active trails using pgmpy for the late-for-school model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.is_active_trail('accident', 'rain')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.is_active_trail('accident', 'rain',\n",
    "                      observed='traffic_jam')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.is_active_trail('getting_up_late', 'rain')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.is_active_trail('getting_up_late', 'rain',\n",
    "                      observed='late_for_school')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Relating graphs and distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* IMAP\n",
    "* IMAP to factorization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will unify these two concepts and show that a probability distribution D can only be represented using a graph G, if and only if D can be represented as a set of CPDs associated with the graph G."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 참고\n",
    "* [10] Independencies in Bayesian Networks - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/2.1.5-Repn-Ind-BNs.pdf\n",
    "* [11] Murpy's Machine Learing: 10. Directed Graphical Model - http://www.slideshare.net/JungkyuLee1/murpys-machine-learing-10-directed-graphical-model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IMAP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A graph object G is called an IMAP of a probability distribution D if the set of independency assertions in G, denoted by I(G), is a subset of the set of independencies in D, denoted by I(D)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.17.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this distribution, we can see that P(X) = 0.5 and P(Y) = 0.5. Also, P(X, Y) = P(X) P(Y). Hence, the two random variables X and Y are independent. If we try to represent any two random variables using a network, we have three possibilities:\n",
    "* A graph with two disconnected nodes X and Y\n",
    "* A graph with an edge from X → Y\n",
    "* A graph with an edge from Y → X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see from the previous distribution that I (D) = {X ⊥ Y}. In the case of disconnected nodes, we also have I (G ) = {X ⊥ Y } , whereas for the other two graphs, we have I(G) = φ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hence, all the three graphs are IMAPS of the distribution, and any of these can be used to represent the probability distribution. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Perepect Map\n",
    "However, the graph with both nodes disconnected is able to best represent the probability distribution and is known as the Perfect Map."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IMAP to factorization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The structure of the Bayesian network encodes the independencies between the random variables, and every probability distribution for which this BN is an IMAP needs to satisfy these independencies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Taking the example of the late-for-school model, using the chain rule, we can show that for any distribution, the joint probability distribution would be as follows:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.18.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, if we consider a distribution for which the BN is an IMAP, we get information about the independencies in the distribution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.19.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CPD representations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Deterministic CPDs\n",
    "* Context-specific CPDs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 참고\n",
    "* [12] stanford-pgm/slides : Structured CPDs - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/Section-2-Representation-CPDs.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Till now, we have only been working with tabular CPDs. In a tabular CPD, we take all the possible combinations of different states of a variable and represent them\n",
    "in a tabular form. However, in many cases, tabular CPD is not the best choice to represent CPDs. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many other cases which we will discuss in this section when other types of representation are a better choice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deterministic CPDs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.20.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### logic gates example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.21.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, X and Y are the inputs to the logic gate and Z is the output. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also see that having a deterministic variable gives up more information about the independencies in the network. If we are given the values of X and Y, we know the value of Z, which leads us to the :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.22.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Context-specific CPDs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Tree CPD\n",
    "* Rule CPD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tree CPD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.23.png\" width=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.24.png\" width=600 />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Now, let's see how we can implement this using pmgpy:\n",
    "from pgmpy.factors import TreeCPD, Factor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tree_cpd = TreeCPD([\n",
    "                      ('B', Factor(['A'], [2], [0.8, 0.2]), '0'),\n",
    "                      ('B', 'C', '1'),\n",
    "                      ('C', Factor(['A'], [2], [0.1, 0.9]), '0'),\n",
    "                      ('C', 'D', '1'),\n",
    "                      ('D', Factor(['A'], [2], [0.9, 0.1]), '0'),\n",
    "                      ('D', Factor(['A'], [2], [0.4, 0.6]), '1')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Rule CPD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rule CPD is another more explicit form of representation of CPDs. Rule CPD is basically a set of rules along with the corresponding values of the variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/cap1.25.png\"  />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Let's see the code implementation using pgmpy:\n",
    "from pgmpy.factors import RuleCPD\n",
    "rule = RuleCPD('A', {('A_0', 'B_0'): 0.8,\n",
    "                                ('A_1', 'B_0'): 0.2,\n",
    "                                ('A_0', 'B_1', 'C_0'): 0.4,\n",
    "                                ('A_1', 'B_1', 'C_0'): 0.6,\n",
    "                                ('A_0', 'B_1', 'C_1'): 0.9,\n",
    "                                ('A_1', 'B_1', 'C_1'): 0.1})\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 참고자료 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "* [1] Mastering Probabilistic Graphical Models using Python - http://www.amazon.com/Mastering-Probabilistic-Graphical-Models-Python/dp/1784394688\n",
    "* [2] Probabilistic Graphical Models - https://www.coursera.org/course/pgm\n",
    "* [3] stanford-pgm/slides/Section-1-Introduction - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/Section-1-Introduction-Combined.pdf\n",
    "* [4] stanford-pgm/slides/1.1.2-Intro-distributions - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/1.1.2-Intro-distributions.pdf\n",
    "* [5] stanford-pgm/slides/1.1.3-Intro-factors - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/1.1.3-Intro-factors.pdf\n",
    "* [6] stanford-pgm/slides/2.1.4-Repn-Ind-conditional-independence - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/2.1.4-Repn-Ind-conditional-independence.pdf\n",
    "* [7] stanford-pgm/slides/2.1.1-Repn-BNs-semantics - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/2.1.1-Repn-BNs-semantics.pdf\n",
    "* [8] stanford-pgm/slides/2.1.2-Repn-BNs-patterns - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/2.1.2-Repn-BNs-patterns.pdf\n",
    "* [9] stanford-pgm/slides/2.1.3-Repn-BNs-flow-influence - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/2.1.3-Repn-BNs-flow-influence.pdf\n",
    "* [10] Independencies in Bayesian Networks - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/2.1.5-Repn-Ind-BNs.pdf\n",
    "* [11] Murpy's Machine Learing: 10. Directed Graphical Model - http://www.slideshare.net/JungkyuLee1/murpys-machine-learing-10-directed-graphical-model\n",
    "* [12] stanford-pgm/slides : Structured CPDs - http://spark-university.s3.amazonaws.com/stanford-pgm/slides/Section-2-Representation-CPDs.pdf"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}