Chapter 20 - Directed Graphs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 20. Directed Graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 20.1 Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Directed graphs are similar to undirected graphs, but there are arrows between vertices instead of edges.  Like undirected graphs, directed graphs can be used to represent independence relations.  They can also be used as an alternative to counterfactuals to represent causal relationships.  Some people use the phrase **Bayesian network** to refer to a directed graph endowed with a probability distribution.  This is a poor choice of terminology.  Statistical inference for directed graphs can be performed using frequentist or Bayesian methods so it is misleading to call them Bayesian networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 20.2 DAG's"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A **directed graph** $\\mathcal{G}$ consists of a set of vertices $V$ and an edge set $E$ of ordered pairs of variables.  If $(X, Y) \\in E$ then there is an arrow pointing from $X$ to $Y$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
       " -->\r\n",
       "<!-- Title: %3 Pages: 1 -->\r\n",
       "<svg width=\"134pt\" height=\"116pt\"\r\n",
       " viewBox=\"0.00 0.00 134.00 116.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 112)\">\r\n",
       "<title>%3</title>\r\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-112 130,-112 130,4 -4,4\"/>\r\n",
       "<!-- Y -->\r\n",
       "<g id=\"node1\" class=\"node\"><title>Y</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"63\" cy=\"-90\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"63\" y=\"-86.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">Y</text>\r\n",
       "</g>\r\n",
       "<!-- X -->\r\n",
       "<g id=\"node2\" class=\"node\"><title>X</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-14.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">X</text>\r\n",
       "</g>\r\n",
       "<!-- Y&#45;&gt;X -->\r\n",
       "<g id=\"edge1\" class=\"edge\"><title>Y&#45;&gt;X</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M54.6504,-72.7646C50.2885,-64.2831 44.8531,-53.7144 39.9587,-44.1974\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"42.9904,-42.4395 35.3043,-35.1473 36.7654,-45.6409 42.9904,-42.4395\"/>\r\n",
       "</g>\r\n",
       "<!-- Z -->\r\n",
       "<g id=\"node3\" class=\"node\"><title>Z</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"99\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"99\" y=\"-14.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">Z</text>\r\n",
       "</g>\r\n",
       "<!-- Y&#45;&gt;Z -->\r\n",
       "<g id=\"edge2\" class=\"edge\"><title>Y&#45;&gt;Z</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M71.3496,-72.7646C75.7115,-64.2831 81.1469,-53.7144 86.0413,-44.1974\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"89.2346,-45.6409 90.6957,-35.1473 83.0096,-42.4395 89.2346,-45.6409\"/>\r\n",
       "</g>\r\n",
       "</g>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x1b3dd2396a0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from graphviz import Digraph\n",
    "\n",
    "d = Digraph()\n",
    "\n",
    "d.edge('Y', 'X')\n",
    "d.edge('Y', 'Z')\n",
    "\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If an arrow connects two variables $X$ and $Y$ (in either direction) we say that $X$ and $Y$ are **adjacent**.  If there is an arrow from $X$ to $Y$ then $X$ is a **parent** of $Y$ and $Y$ is a **child** of $X$.  The set of all parents of $X$ is denoted by $\\pi_X$ or $\\pi(X)$.  A **directed path** from $X$ to $Y$ is a set of vertices beginning with $X$, ending with $Y$ such that each pair is connected by an arrow and all of the arrows point in the same direction:\n",
    "\n",
    "$$ X \\rightarrow \\cdots \\rightarrow Y\n",
    "\\quad \\text{or} \\quad\n",
    "X \\leftarrow \\cdots \\leftarrow Y $$\n",
    "\n",
    "A sequence of adjacent vertices starting with $X$ and ending with $Y$ but ignoring the directions of the arrows is called an **undirected path**.  $X$ is an **ancestor** of $Y$ if there is a directed path from $X$ to $Y$.  We also say that $Y$ is a **descendant** of $X$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A configuration of the form:\n",
    "\n",
    "$$ X \\rightarrow Y \\leftarrow Z $$\n",
    "\n",
    "is called a **collider**.  A configuration not of that form is called a **non-collider**, for example,\n",
    "\n",
    "$$ X \\rightarrow Y \\rightarrow Z\n",
    "\\quad \\text{or} \\quad\n",
    "X \\leftarrow Y \\leftarrow Z$$\n",
    "\n",
    "A directed path that starts and ends at the same variable is called a **cycle**.  A directed graph is **acyclic** if it has no cycles.  In this case we say that the graph is a **directed acyclic graph** or **DAG**.  From now on, we will only deal with graphs that are DAG's."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 20.3 Probability and DAG's"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $\\mathcal{G}$ be a DAG with vertices $V = (X_1, \\dots, X_k)$.\n",
    "\n",
    "If $\\mathbb{P}$ is a distribution for $V$ with probability function $p$, we say that **$\\mathbb{P}$ is Markov to $\\mathcal{G}$** or that **$\\mathcal{G} represents $\\mathbb{P}$** if\n",
    "\n",
    "$$ p(v) = \\prod_{i=1}^k p(x_i | \\pi_i) $$\n",
    "\n",
    "where the $\\pi_i$ are the parents of $X_i$.  The set of distributions represented by $\\mathcal{G}$ is denoted by $M(\\mathcal{G})$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following theorem says that $\\mathbb{P} \\in M(\\mathcal{G})$ if and only if the following **Markov Condition** holds.  Roughly speaking, the Markov Condition says that every variable $W$ is independent of the \"past\" given its parents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 20.3**. A distribution $\\mathbb{P} \\in M(\\mathcal{G})$ if and only if the following **Markov Condition** holds: for every variable $W$,\n",
    "\n",
    "$$W \\text{ ⫫ } \\overline{W} | \\pi_W $$\n",
    "\n",
    "where $\\overline{W}$ denotes all the other variables except parents and descendants of $W$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 20.4 More Independence Relations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Markov Condition allows us to list some independence relations.  These relations may logically imply other independence relations.  Consider this DAG:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
       " -->\r\n",
       "<!-- Title: %3 Pages: 1 -->\r\n",
       "<svg width=\"134pt\" height=\"188pt\"\r\n",
       " viewBox=\"0.00 0.00 134.00 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\r\n",
       "<title>%3</title>\r\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-184 130,-184 130,4 -4,4\"/>\r\n",
       "<!-- X2 -->\r\n",
       "<g id=\"node1\" class=\"node\"><title>X2</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-162\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-158.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">X2</text>\r\n",
       "</g>\r\n",
       "<!-- X3 -->\r\n",
       "<g id=\"node2\" class=\"node\"><title>X3</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-90\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-86.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">X3</text>\r\n",
       "</g>\r\n",
       "<!-- X2&#45;&gt;X3 -->\r\n",
       "<g id=\"edge1\" class=\"edge\"><title>X2&#45;&gt;X3</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M27,-143.697C27,-135.983 27,-126.712 27,-118.112\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"30.5001,-118.104 27,-108.104 23.5001,-118.104 30.5001,-118.104\"/>\r\n",
       "</g>\r\n",
       "<!-- X5 -->\r\n",
       "<g id=\"node5\" class=\"node\"><title>X5</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"63\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"63\" y=\"-14.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">X5</text>\r\n",
       "</g>\r\n",
       "<!-- X3&#45;&gt;X5 -->\r\n",
       "<g id=\"edge4\" class=\"edge\"><title>X3&#45;&gt;X5</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M35.3496,-72.7646C39.7115,-64.2831 45.1469,-53.7144 50.0413,-44.1974\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"53.2346,-45.6409 54.6957,-35.1473 47.0096,-42.4395 53.2346,-45.6409\"/>\r\n",
       "</g>\r\n",
       "<!-- X1 -->\r\n",
       "<g id=\"node3\" class=\"node\"><title>X1</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"99\" cy=\"-162\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"99\" y=\"-158.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">X1</text>\r\n",
       "</g>\r\n",
       "<!-- X1&#45;&gt;X3 -->\r\n",
       "<g id=\"edge2\" class=\"edge\"><title>X1&#45;&gt;X3</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M84.4297,-146.834C74.2501,-136.938 60.4761,-123.546 48.9694,-112.359\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"51.4055,-109.846 41.7957,-105.385 46.5259,-114.865 51.4055,-109.846\"/>\r\n",
       "</g>\r\n",
       "<!-- X4 -->\r\n",
       "<g id=\"node4\" class=\"node\"><title>X4</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"99\" cy=\"-90\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"99\" y=\"-86.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">X4</text>\r\n",
       "</g>\r\n",
       "<!-- X1&#45;&gt;X4 -->\r\n",
       "<g id=\"edge3\" class=\"edge\"><title>X1&#45;&gt;X4</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M99,-143.697C99,-135.983 99,-126.712 99,-118.112\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"102.5,-118.104 99,-108.104 95.5001,-118.104 102.5,-118.104\"/>\r\n",
       "</g>\r\n",
       "<!-- X4&#45;&gt;X5 -->\r\n",
       "<g id=\"edge5\" class=\"edge\"><title>X4&#45;&gt;X5</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M90.6504,-72.7646C86.2885,-64.2831 80.8531,-53.7144 75.9587,-44.1974\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"78.9904,-42.4395 71.3043,-35.1473 72.7654,-45.6409 78.9904,-42.4395\"/>\r\n",
       "</g>\r\n",
       "</g>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x1b3dd26f4e0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from graphviz import Digraph\n",
    "\n",
    "d = Digraph()\n",
    "\n",
    "d.edge('X2', 'X3')\n",
    "d.edge('X1', 'X3')\n",
    "d.edge('X1', 'X4')\n",
    "d.edge('X3', 'X5')\n",
    "d.edge('X4', 'X5')\n",
    "\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Markov Condition implies:\n",
    "\n",
    "- $X_1 \\text{ ⫫ } X_2$\n",
    "- $X_2 \\text{ ⫫ } \\{ X_1, X_4 \\}$\n",
    "- $X_3 \\text{ ⫫ } X_4 | \\{ X_1, X_2 \\}$\n",
    "- $X_4 \\text{ ⫫ } \\{ X_2, X_3 \\} | X_1$\n",
    "- $X_5 \\text{ ⫫ } \\{ X_1, X_2 \\} | \\{X_3, X_4\\}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It turns out that these conditions imply:\n",
    "\n",
    "$$ \\{ X_4, X_5 \\} \\text{ ⫫ } X_2 | \\{ X_1, X_3 \\} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do we find these extra independence relations?  The answer is **d-separation**, which can be summarized by 3 rules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The rules of d-separation**\n",
    "\n",
    "1.  In a non-collider $(X, Y, Z)$, $X$ and $Z$ are **d-connected**, but they are **d-separated** given $Y$.\n",
    "2.  If $X$ and $Z$ collide at $Y$ then $X$ and $Z$ are **d-separated** but they are **d-connected** given $Y$.\n",
    "3.  Conditioning on the descendant of a collider has the same effect as conditioning on the collider.  Thus in the figure below, $X$ and $Z$ are **d-separated** but they are **d-connected** given $W$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
       " -->\r\n",
       "<!-- Title: %3 Pages: 1 -->\r\n",
       "<svg width=\"134pt\" height=\"188pt\"\r\n",
       " viewBox=\"0.00 0.00 134.00 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\r\n",
       "<title>%3</title>\r\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-184 130,-184 130,4 -4,4\"/>\r\n",
       "<!-- X -->\r\n",
       "<g id=\"node1\" class=\"node\"><title>X</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-162\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-158.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">X</text>\r\n",
       "</g>\r\n",
       "<!-- Y -->\r\n",
       "<g id=\"node2\" class=\"node\"><title>Y</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"63\" cy=\"-90\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"63\" y=\"-86.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">Y</text>\r\n",
       "</g>\r\n",
       "<!-- X&#45;&gt;Y -->\r\n",
       "<g id=\"edge1\" class=\"edge\"><title>X&#45;&gt;Y</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M35.3496,-144.765C39.7115,-136.283 45.1469,-125.714 50.0413,-116.197\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"53.2346,-117.641 54.6957,-107.147 47.0096,-114.439 53.2346,-117.641\"/>\r\n",
       "</g>\r\n",
       "<!-- W -->\r\n",
       "<g id=\"node4\" class=\"node\"><title>W</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"63\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"63\" y=\"-14.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">W</text>\r\n",
       "</g>\r\n",
       "<!-- Y&#45;&gt;W -->\r\n",
       "<g id=\"edge3\" class=\"edge\"><title>Y&#45;&gt;W</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M63,-71.6966C63,-63.9827 63,-54.7125 63,-46.1124\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"66.5001,-46.1043 63,-36.1043 59.5001,-46.1044 66.5001,-46.1043\"/>\r\n",
       "</g>\r\n",
       "<!-- Z -->\r\n",
       "<g id=\"node3\" class=\"node\"><title>Z</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"99\" cy=\"-162\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"99\" y=\"-158.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">Z</text>\r\n",
       "</g>\r\n",
       "<!-- Z&#45;&gt;Y -->\r\n",
       "<g id=\"edge2\" class=\"edge\"><title>Z&#45;&gt;Y</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M90.6504,-144.765C86.2885,-136.283 80.8531,-125.714 75.9587,-116.197\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"78.9904,-114.439 71.3043,-107.147 72.7654,-117.641 78.9904,-114.439\"/>\r\n",
       "</g>\r\n",
       "</g>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x1b3dd26f208>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from graphviz import Digraph\n",
    "\n",
    "d = Digraph()\n",
    "\n",
    "d.edge('X', 'Y')\n",
    "d.edge('Z', 'Y')\n",
    "d.edge('Y', 'W')\n",
    "\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a more formal definition of d-separation.  Let $X$ and $Y$ be distinct vertices and let $W$ be a set of vertices not containing $X$ or $Y$.  Then $X$ and $Y$ are **d-separated given $W$** if there exists no undirected path $U$ between $X$ and $Y$ such that (i) every collider on $U$ has a descendant in $W$ and (ii) no other vertex on $U$ is in $W$.  If $U$, $V$, $W$ are distinct sets of vertices and $U$ and $V$ are not empty, then $U$ and $V$ are d-separated given $W$ if for every $X \\in U$ and $Y \\in V$, $X$ and $Y$ are d-separated given $W$.  Vertices that are not d-separated are said to be d-connected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 20.7 (Spirtes, Glymour and Scheines)**.  Let $A$, $B$ and $C$ be disjoint sets of vertices.  Then $A \\text{ ⫫ } B | C$ if and only if $A$ and $B$ are d-separated by $C$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Graphs that look different may imply the same independence relations.  If $\\mathcal{G}$ is a DAG, we let $\\mathcal{I}(\\mathcal{G})$ denote all the independence statements implied by $\\mathcal{G}$.  Two DAG's $\\mathcal{G}_1$ and $\\mathcal{G}_2$ for the same variables $V$ are **Markov equivalent** if $\\mathcal{I}(\\mathcal{G}_1) = \\mathcal{I}(\\mathcal{G}_2)$.  Given a DAG $\\mathcal{G}$, let $\\text{skeleton}(\\mathcal{G})$ denote the undirected graph obtained by replacing the arrows with undirected edges."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 20.9**.  Two DAG's $\\mathcal{G}_1$ and $\\mathcal{G}_2$ are Markov equivalent if and only if (i) $\\text{skeleton}(\\mathcal{G}_1) = \\text{skeleton}(\\mathcal{G}_2)$ and (ii) $\\mathcal{G}_1$ and $\\mathcal{G}_2$ have the same colliders."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 20.5 Estimation for DAG's"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $\\mathcal{G}$ be a DAG.  Assume that all variables $V = \\{ X_1, \\dots, X_m \\}$ are discrete.  The probability function can be written\n",
    "\n",
    "$$ p(v) = \\prod_{i=1}^m p(x_i | \\pi_i) $$\n",
    "\n",
    "To estimate $p(v)$ we need to estimate $p(x_i | \\pi_i)$ for each $i$.  Think of the parents of $X_i$ as one discrete variable $\\overline{X}_i$ with many levels.  For example, suppose that $X_3$ has parents $X_1$ and $X_2$, and that $X_1 \\in \\{ 0, 1, 2 \\}$ while $X_2 \\in \\{ 0, 1 \\}$.  We can regard the parents $X_1$ and $X_2$ as a single variable $\\overline{X}_3$ defined by\n",
    "\n",
    "$$ \\overline{X}_3 = \\begin{cases}\n",
    "1 & \\text{if } X_1 = 0, X_2 = 0\\\\\n",
    "2 & \\text{if } X_1 = 0, X_2 = 1\\\\\n",
    "3 & \\text{if } X_1 = 1, X_2 = 0\\\\\n",
    "4 & \\text{if } X_1 = 1, X_2 = 1\\\\\n",
    "5 & \\text{if } X_1 = 2, X_2 = 0\\\\\n",
    "6 & \\text{if } X_1 = 2, X_2 = 1\\\\\n",
    "\\end{cases}$$\n",
    "\n",
    "Hence we can write\n",
    "\n",
    "$$ p(v) = \\prod_{i=1}^m p(x_i | \\overline{x}_i) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem 20.11**.  Let $V_1, \\dots, V_n$ be IID random vectors from distribution $p$ where\n",
    "\n",
    "$$ p(v) = \\prod_{i=1}^k p(x_i | \\overline{x}_i) $$\n",
    "\n",
    "The maximum likelihood estimator of $p$ is\n",
    "\n",
    "$$ \\hat{p}(v) = \\prod_{i=1}^k \\hat{p}(x_i | \\overline{x}_i) $$\n",
    "\n",
    "where\n",
    "\n",
    "$$ \\hat{p}(x_i | \\overline{x}_i) = \\frac{\\# \\{i : X_i = x_i \\text{ and } \\overline{X}_i = \\overline{x}_i \\}}{\\# \\{i : Y_i = y \\}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is possible to extend these ideas to continuous random variables as well.  For example, we might use some parametric model $p(x | \\pi_x; \\theta_x)$ for each conditional density.  The likelihood function is then\n",
    "\n",
    "$$ \\mathcal{L}(\\theta) = \\prod_{i=1}^n p(V_i; \\theta) = \\prod_{i=1}^n \\prod_{j=1}^m p(X_{ij} | \\pi_j; \\theta_j) $$\n",
    "\n",
    "where $X_{ij}$ is the value of $X_j$ for the $i$-th data point and $\\theta_j$ are the parameters for the $j$-th conditional density.  We can then proceed using maximum likelihood."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, we have assumed that the structure of the DAG is given.  One can also try to estimate the structure of the DAG itself from the data.  However, there are many possible DAG's so you would need much data for such a method to be reliable.  Producing a valid, accurate confidence set for the DAG structure would require astronomical sample sizes.  DAG's are thus most useful for encoding conditional independence information rather than discovering it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 20.6 Causation Revisited"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We discussed causation in Chapter 19 using the idea of counterfactual random variables.  A different approach uses DAG's.  The two approaches are mathematically equivalent though they appear to be quite different.  The extra element in DAG is the idea of **intervention**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
       " -->\r\n",
       "<!-- Title: %3 Pages: 1 -->\r\n",
       "<svg width=\"89pt\" height=\"188pt\"\r\n",
       " viewBox=\"0.00 0.00 89.00 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\r\n",
       "<title>%3</title>\r\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-184 85,-184 85,4 -4,4\"/>\r\n",
       "<!-- X -->\r\n",
       "<g id=\"node1\" class=\"node\"><title>X</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"54\" cy=\"-162\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"54\" y=\"-158.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">X</text>\r\n",
       "</g>\r\n",
       "<!-- Y -->\r\n",
       "<g id=\"node2\" class=\"node\"><title>Y</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-90\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-86.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">Y</text>\r\n",
       "</g>\r\n",
       "<!-- X&#45;&gt;Y -->\r\n",
       "<g id=\"edge1\" class=\"edge\"><title>X&#45;&gt;Y</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47.6014,-144.411C44.4864,-136.335 40.6663,-126.431 37.1654,-117.355\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"40.4045,-116.027 33.5403,-107.956 33.8735,-118.546 40.4045,-116.027\"/>\r\n",
       "</g>\r\n",
       "<!-- Z -->\r\n",
       "<g id=\"node3\" class=\"node\"><title>Z</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"54\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"54\" y=\"-14.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">Z</text>\r\n",
       "</g>\r\n",
       "<!-- X&#45;&gt;Z -->\r\n",
       "<g id=\"edge3\" class=\"edge\"><title>X&#45;&gt;Z</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M57.6538,-143.908C59.6758,-133.569 61.9808,-120.09 63,-108 64.3441,-92.0566 64.3441,-87.9434 63,-72 62.2834,-63.4991 60.9311,-54.3119 59.4884,-46.0122\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"62.9142,-45.2892 57.6538,-36.0925 56.0309,-46.5623 62.9142,-45.2892\"/>\r\n",
       "</g>\r\n",
       "<!-- Y&#45;&gt;Z -->\r\n",
       "<g id=\"edge2\" class=\"edge\"><title>Y&#45;&gt;Z</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M33.3986,-72.411C36.5136,-64.3352 40.3337,-54.4312 43.8346,-45.3547\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47.1265,-46.5458 47.4597,-35.9562 40.5955,-44.0267 47.1265,-46.5458\"/>\r\n",
       "</g>\r\n",
       "</g>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x1b3dd26f7b8>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from graphviz import Digraph\n",
    "\n",
    "d = Digraph()\n",
    "\n",
    "d.edge('X', 'Y')\n",
    "d.edge('Y', 'Z')\n",
    "d.edge('X', 'Z')\n",
    "\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider the DAG above.  The probability function for a distribution consistent with it has the form $p(x, y, z) = p(x)p(y | x)p(z| x, y)$.  Here is pseudo-code for generating from this distribution:\n",
    "\n",
    "```\n",
    "for i = 1, ..., n:\n",
    "  x_i <- p_X(x_i)\n",
    "  y_i <- p_{Y | X}(y_i | x_i)\n",
    "  z_i <- p_{Z | X, Y}(z_i | x_i, y_i)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we repeat this code many times yielding data $(x_1, y_1, z_1), \\dots, (x_n, y_n, z_n)$.  Among all the times that we observe $Y = y$, how often is $Z = z$?  The answer to this question is given by the conditional distribution of $Z | Y$.\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbb{P}(Z = z | Y = y) &= \\frac{\\mathbb{P}(Y = y, Z = z)}{\\mathbb{P}(Y = y)} = \\frac{p(y, z)}{p(y)} \\\\\n",
    "&= \\frac{\\sum_x p(x, y, z)}{p(y)} = \\frac{\\sum_x p(x) p(y | x) p(z | x, y)}{p(y)} \\\\\n",
    "&= \\sum_x p(z | x, y) \\frac{p(y | x) p(x)}{p(y)} = \\sum_x p(z | x, y) \\frac{p(x, y)}{p(y)} \\\\\n",
    "&= \\sum_x p(z | x, y) p(x | y)\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now suppose we **intervene** by changing the computer code.  Specifically, suppose we fix $Y$ at the value $y$.  The code now looks like this:\n",
    "\n",
    "```\n",
    "set Y = y\n",
    "for i = 1, ..., n:\n",
    "  x_i <- p_X(x_i)\n",
    "  z_i <- p_{Z | X, Y}(z_i | x_i, y)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having **set** $Y = y$, how often was $Z = z$?  To answer, note that the intervention has changed the joint probability to be \n",
    "\n",
    "$$ p^*(x, z) = p(x) p(z | x, y) $$\n",
    "\n",
    "The answer to our question is given by the marginal distribution\n",
    "\n",
    "$$ p^*(z) = \\sum_x p^*(x, z) = \\sum_x p(x) p(z | x, y) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We shall denote this as $\\mathbb{P}(Z = z | Y := y)$ or $p(z | Y := y)$.  We call $\\mathbb{P}(Z = z | Y = y)$ **conditioning by observation** or **passive conditioning**.  We call $\\mathbb{P}(Z = z | Y := y)$ **conditioning by intervention** or **active conditioning**.\n",
    "\n",
    "- Passive conditioning is used to answer a predictive question like \"Given that Joe smokes, what is the probability that he will get lung cancer?\".\n",
    "- Active conditioning is used to answer a predictive question like \"If Joe quits smoking, what is the probability that he will get lung cancer?\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider a pair $(\\mathcal{G}, \\mathbb{P})$ where $\\mathcal{G}$ is a DAG and $\\mathbb{P}$ is a distribution for the variables $V$ of the DAG.  Let $p$ denote the probability function for $p$.  Consider intervening and fixing a variable $X$ to be equal to $x$.  We represent the intervention by doing two things:\n",
    "\n",
    "1.  Create a new DAG $\\mathcal{G}^*$ by removing all arrows pointing into $X$;\n",
    "2.  Create a new distribution $\\mathbb{P}^*$ with probability function $p^*(v) = \\mathbb{P}(V = v | X := x)$ by removing the term $p(x | \\pi_X)$ from $p(v)$.\n",
    "\n",
    "The new pair $(\\mathcal{G}^*, \\mathbb{P}^*)$ represents the intervention \"set $X = x$\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use DAG's to represent confounding variables.  If $X$ is a treatment and $Y$ is an outcome, a confounding variable $Z$ is a variable with arrows into both $X$ and $Y$.  It is easy to check, using the formalism of interventions, that the following facts are true.\n",
    "\n",
    "- In a randomized study, the arrow between $Z$ and $X$ is broken.  In this case, even with $Z$ unobserved, the causal relationship between $X$ and $Y$ is estimable because it can be shown that $\\mathbb{E}(Y | X := x) = \\mathbb{E}(Y | X = x)$ which does not involved the unobserved $Z$.  \n",
    "- In an observational study, with all confounders observed, we again get $\\mathbb{E}(Y | X := x) = \\int \\mathbb{E}(Y | X = x, Z = z) d F_Z(z)$ as a formula for the expectation.  if $Z$ is unobserved then we cannot estimate the causal effect because this expectation involves the unobserved $Z$.  $\\mathbb{P}(Y = y | X = x) \\neq \\mathbb{P}(Y = y | X := x)$ which is just another way of saying that causation is not association."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, we can make a precise connection between DAG's and counterfactuals as follows.  Suppose that $X$ and $Y$ are binary.  Define the confounding variable $Z$ by\n",
    "\n",
    "$$ Z = \\begin{cases}\n",
    "1 & \\text{if } (C_0, C_1) = (0, 0) \\\\\n",
    "2 & \\text{if } (C_0, C_1) = (0, 1) \\\\\n",
    "3 & \\text{if } (C_0, C_1) = (1, 0) \\\\\n",
    "4 & \\text{if } (C_0, C_1) = (1, 1)\n",
    "\\end{cases}$$\n",
    "\n",
    "From this approach, one can make the correspondence between the DAG and counterfactual approaches explicit.  This is left as an exercise for the reader."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 20.8 Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 20.8.1**. Consider the three DAG's below without a collider.  Prove that $X \\text{ ⫫ } Z | Y$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
       " -->\r\n",
       "<!-- Title: %3 Pages: 1 -->\r\n",
       "<svg width=\"62pt\" height=\"188pt\"\r\n",
       " viewBox=\"0.00 0.00 62.00 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\r\n",
       "<title>%3</title>\r\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-184 58,-184 58,4 -4,4\"/>\r\n",
       "<!-- X -->\r\n",
       "<g id=\"node1\" class=\"node\"><title>X</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-162\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-158.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">X</text>\r\n",
       "</g>\r\n",
       "<!-- Y -->\r\n",
       "<g id=\"node2\" class=\"node\"><title>Y</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-90\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-86.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">Y</text>\r\n",
       "</g>\r\n",
       "<!-- X&#45;&gt;Y -->\r\n",
       "<g id=\"edge1\" class=\"edge\"><title>X&#45;&gt;Y</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M27,-143.697C27,-135.983 27,-126.712 27,-118.112\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"30.5001,-118.104 27,-108.104 23.5001,-118.104 30.5001,-118.104\"/>\r\n",
       "</g>\r\n",
       "<!-- Z -->\r\n",
       "<g id=\"node3\" class=\"node\"><title>Z</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-14.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">Z</text>\r\n",
       "</g>\r\n",
       "<!-- Y&#45;&gt;Z -->\r\n",
       "<g id=\"edge2\" class=\"edge\"><title>Y&#45;&gt;Z</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M27,-71.6966C27,-63.9827 27,-54.7125 27,-46.1124\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"30.5001,-46.1043 27,-36.1043 23.5001,-46.1044 30.5001,-46.1043\"/>\r\n",
       "</g>\r\n",
       "</g>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x1b3dd26f518>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from graphviz import Digraph\n",
    "\n",
    "a = Digraph()\n",
    "a.edge('X', 'Y')\n",
    "a.edge('Y', 'Z')\n",
    "\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
       " -->\r\n",
       "<!-- Title: %3 Pages: 1 -->\r\n",
       "<svg width=\"62pt\" height=\"188pt\"\r\n",
       " viewBox=\"0.00 0.00 62.00 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\r\n",
       "<title>%3</title>\r\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-184 58,-184 58,4 -4,4\"/>\r\n",
       "<!-- Y -->\r\n",
       "<g id=\"node1\" class=\"node\"><title>Y</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-90\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-86.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">Y</text>\r\n",
       "</g>\r\n",
       "<!-- X -->\r\n",
       "<g id=\"node2\" class=\"node\"><title>X</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-14.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">X</text>\r\n",
       "</g>\r\n",
       "<!-- Y&#45;&gt;X -->\r\n",
       "<g id=\"edge1\" class=\"edge\"><title>Y&#45;&gt;X</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M27,-71.6966C27,-63.9827 27,-54.7125 27,-46.1124\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"30.5001,-46.1043 27,-36.1043 23.5001,-46.1044 30.5001,-46.1043\"/>\r\n",
       "</g>\r\n",
       "<!-- Z -->\r\n",
       "<g id=\"node3\" class=\"node\"><title>Z</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-162\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-158.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">Z</text>\r\n",
       "</g>\r\n",
       "<!-- Z&#45;&gt;Y -->\r\n",
       "<g id=\"edge2\" class=\"edge\"><title>Z&#45;&gt;Y</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M27,-143.697C27,-135.983 27,-126.712 27,-118.112\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"30.5001,-118.104 27,-108.104 23.5001,-118.104 30.5001,-118.104\"/>\r\n",
       "</g>\r\n",
       "</g>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x1b3dd279048>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from graphviz import Digraph\n",
    "\n",
    "b = Digraph()\n",
    "b.edge('Y', 'X')\n",
    "b.edge('Z', 'Y')\n",
    "\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
       " -->\r\n",
       "<!-- Title: %3 Pages: 1 -->\r\n",
       "<svg width=\"134pt\" height=\"116pt\"\r\n",
       " viewBox=\"0.00 0.00 134.00 116.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 112)\">\r\n",
       "<title>%3</title>\r\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-112 130,-112 130,4 -4,4\"/>\r\n",
       "<!-- Y -->\r\n",
       "<g id=\"node1\" class=\"node\"><title>Y</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"63\" cy=\"-90\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"63\" y=\"-86.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">Y</text>\r\n",
       "</g>\r\n",
       "<!-- X -->\r\n",
       "<g id=\"node2\" class=\"node\"><title>X</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-14.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">X</text>\r\n",
       "</g>\r\n",
       "<!-- Y&#45;&gt;X -->\r\n",
       "<g id=\"edge1\" class=\"edge\"><title>Y&#45;&gt;X</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M54.6504,-72.7646C50.2885,-64.2831 44.8531,-53.7144 39.9587,-44.1974\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"42.9904,-42.4395 35.3043,-35.1473 36.7654,-45.6409 42.9904,-42.4395\"/>\r\n",
       "</g>\r\n",
       "<!-- Z -->\r\n",
       "<g id=\"node3\" class=\"node\"><title>Z</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"99\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"99\" y=\"-14.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">Z</text>\r\n",
       "</g>\r\n",
       "<!-- Y&#45;&gt;Z -->\r\n",
       "<g id=\"edge2\" class=\"edge\"><title>Y&#45;&gt;Z</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M71.3496,-72.7646C75.7115,-64.2831 81.1469,-53.7144 86.0413,-44.1974\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"89.2346,-45.6409 90.6957,-35.1473 83.0096,-42.4395 89.2346,-45.6409\"/>\r\n",
       "</g>\r\n",
       "</g>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x1b3dd279630>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from graphviz import Digraph\n",
    "\n",
    "c = Digraph()\n",
    "c.edge('Y', 'X')\n",
    "c.edge('Y', 'Z')\n",
    "\n",
    "c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "Note that for all three graphs $X$ and $Z$ are d-separated by $Y$ -- the only undirected path $(X, Y, Z)$ has no colliders, and all its vertices other than $X$ and $Z$ (i.e. $\\{ Y \\}$) are listed in the d-separation condition.  Thus, conditions (i) and (ii) on the definition of d-separation holds.\n",
    "\n",
    "Therefore, from Theorem 20.7, $X \\text{ ⫫ } Z | Y$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 20.8.2**.  Consider the DAG below.  Prove that $X \\text{ ⫫ } Z$ and that $X$ and $Y$ are dependent given $Y$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
       " -->\r\n",
       "<!-- Title: %3 Pages: 1 -->\r\n",
       "<svg width=\"134pt\" height=\"116pt\"\r\n",
       " viewBox=\"0.00 0.00 134.00 116.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 112)\">\r\n",
       "<title>%3</title>\r\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-112 130,-112 130,4 -4,4\"/>\r\n",
       "<!-- Y -->\r\n",
       "<g id=\"node1\" class=\"node\"><title>Y</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"63\" cy=\"-90\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"63\" y=\"-86.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">Y</text>\r\n",
       "</g>\r\n",
       "<!-- X -->\r\n",
       "<g id=\"node2\" class=\"node\"><title>X</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-14.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">X</text>\r\n",
       "</g>\r\n",
       "<!-- Y&#45;&gt;X -->\r\n",
       "<g id=\"edge1\" class=\"edge\"><title>Y&#45;&gt;X</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M54.6504,-72.7646C50.2885,-64.2831 44.8531,-53.7144 39.9587,-44.1974\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"42.9904,-42.4395 35.3043,-35.1473 36.7654,-45.6409 42.9904,-42.4395\"/>\r\n",
       "</g>\r\n",
       "<!-- Z -->\r\n",
       "<g id=\"node3\" class=\"node\"><title>Z</title>\r\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"99\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"99\" y=\"-14.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">Z</text>\r\n",
       "</g>\r\n",
       "<!-- Y&#45;&gt;Z -->\r\n",
       "<g id=\"edge2\" class=\"edge\"><title>Y&#45;&gt;Z</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M71.3496,-72.7646C75.7115,-64.2831 81.1469,-53.7144 86.0413,-44.1974\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"89.2346,-45.6409 90.6957,-35.1473 83.0096,-42.4395 89.2346,-45.6409\"/>\r\n",
       "</g>\r\n",
       "</g>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x1b3dd279588>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from graphviz import Digraph\n",
    "\n",
    "d = Digraph()\n",
    "d.edge('Y', 'X')\n",
    "d.edge('Y', 'Z')\n",
    "\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "Note that $X$ and $Z$ are d-connected by $Y$ -- the undirected path $(X, Y, Z)$ has one collider containing $Y$, so the definition of d-separation does not hold.  Therefore, from Theorem 20.7, $X$ and $Z$ are dependent given $Y$.\n",
    "\n",
    "On the other hand, note that $X$ and $Z$ are d-separated given the empty set -- the only path from $X$ to $Z$ goes through $Y$, which is not in the empty set.  Thus, from Theorem 20.7, $X \\text{ ⫫ } Z$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 20.8.3**.  Let $X \\in \\{0, 1\\}$, $Y \\in \\{0, 1\\}$, $Z \\in \\{ 0, 1, 2 \\}$.  Suppose the distribution of $(X, Y, Z)$ is Markov to:\n",
    "\n",
    "$$ X \\longrightarrow Y \\longrightarrow Z $$\n",
    "\n",
    "Create a joint distribution $p(x, y, z)$ that is Markov to this DAG.  Generate 1000 random vectors from this distribution.  Estimate the distribution from the data using maximum likelihood.  Compare the estimated distribution to the true distribution.  Let $\\theta = (\\theta_{000}, \\theta_{001}, \\dots, \\theta_{112})$ where $\\theta_{rst} = \\mathbb{P}(X = r, Y = s, Z = t)$.  Use the bootstrap to get the standard errors and 95% confidence interval for these 12 parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.  Let's create the following distributions:\n",
    "\n",
    "$$ p_X(x) = \\begin{cases}\n",
    "1/2 & \\text{if } x \\in \\{0, 1\\} \\\\\n",
    "0 & \\text{otherwise}\n",
    "\\end{cases}\n",
    "\\quad\n",
    "p_{Y | X}(y | x) = \\begin{cases}\n",
    "3/4 & \\text{if } y = x \\\\\n",
    "1/4 & \\text{if } x + y = 1 \\\\\n",
    "0 & \\text{otherwise}\n",
    "\\end{cases}\n",
    "\\quad\n",
    "p_{Z | Y}(z | y) = \\begin{cases}\n",
    "1/2 &\\text{if } z = y \\text{ and } z \\in \\{ 0, 1, 2 \\}\\\\\n",
    "1/4 &\\text{if } z \\neq y \\text{ and } z \\in \\{ 0, 1, 2 \\} \\\\\n",
    "0 &\\text{otherwise}\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "and a joint distribution\n",
    "\n",
    "$$ p(x, y, z) = p_X(x) p_{Y | X}(y | x) p_{Z | Y}(z | y) $$\n",
    "\n",
    "By construction, this distribution is Markov to the given DAG.\n",
    "\n",
    "More explicitly, the joint probability distribution is:\n",
    "\n",
    "$$\n",
    "\\begin{array}{ccc|c}\n",
    "X & Y & Z & p \\\\\n",
    "\\hline\n",
    "0 & 0 & 0 & .18750 \\\\\n",
    "0 & 0 & 1 & .09375 \\\\\n",
    "0 & 0 & 2 & .09375 \\\\\n",
    "0 & 1 & 0 & .03125 \\\\\n",
    "0 & 1 & 1 & .06250 \\\\\n",
    "0 & 1 & 2 & .03125 \\\\\n",
    "1 & 0 & 0 & .06250 \\\\\n",
    "1 & 0 & 1 & .03125 \\\\\n",
    "1 & 0 & 2 & .03125 \\\\\n",
    "1 & 1 & 0 & .09375 \\\\\n",
    "1 & 1 & 1 & .18750 \\\\\n",
    "1 & 1 & 2 & .09375 \\\\\n",
    "\\end{array}\n",
    "$$\n",
    "\n",
    "The maximum likelihood estimator of $p$ is\n",
    "\n",
    "$$ \\hat{p}(v) = \\prod_{i=1}^k \\hat{p}(x_i | \\overline{x}_i) $$\n",
    "\n",
    "where\n",
    "\n",
    "$$ \\hat{p}(x_i | \\overline{x}_i) = \\frac{\\# \\{i : X_i = x_i \\text{ and } \\overline{X}_i = \\overline{x}_i \\}}{\\# \\{i : Y_i = y \\}} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def generate_samples(n):\n",
    "    seeds = np.random.uniform(low=0, high=1, size=(n, 3))\n",
    "    result = np.zeros((n, 3), dtype=int)\n",
    "    result[:, 0] = seeds[:, 0] < 1/2\n",
    "    result[:, 1] = np.where(seeds[:, 1] < 3/4, result[:, 0], 1 - result[:, 0])\n",
    "    result[:, 2] = np.where(seeds[:, 2] < 1/2, result[:, 1], (result[:, 1] + np.where(seeds[:, 2] < 3/4, 1, 2)) % 3)\n",
    "    return result\n",
    "\n",
    "def estimate_parameters(X):\n",
    "    n = X.shape[0]\n",
    "    \n",
    "    p_hat_x0 = np.sum(X[:, 0] == 0) / n\n",
    "    p_hat_x1 = np.sum(X[:, 0] == 1) / n\n",
    "    \n",
    "    p_hat_y0_x0 = np.sum((X[:, 0] == 0) & (X[:, 1] == 0)) / np.sum(X[:, 0] == 0)\n",
    "    p_hat_y1_x0 = np.sum((X[:, 0] == 0) & (X[:, 1] == 1)) / np.sum(X[:, 0] == 0)\n",
    "    \n",
    "    p_hat_y0_x1 = np.sum((X[:, 0] == 1) & (X[:, 1] == 0)) / np.sum(X[:, 0] == 1)\n",
    "    p_hat_y1_x1 = np.sum((X[:, 0] == 1) & (X[:, 1] == 1)) / np.sum(X[:, 0] == 1)\n",
    "    \n",
    "    p_hat_z0_y0 = np.sum((X[:, 2] == 0) & (X[:, 1] == 0)) / np.sum(X[:, 1] == 0)\n",
    "    p_hat_z1_y0 = np.sum((X[:, 2] == 1) & (X[:, 1] == 0)) / np.sum(X[:, 1] == 0)\n",
    "    p_hat_z2_y0 = np.sum((X[:, 2] == 2) & (X[:, 1] == 0)) / np.sum(X[:, 1] == 0)\n",
    "\n",
    "    p_hat_z0_y1 = np.sum((X[:, 2] == 0) & (X[:, 1] == 1)) / np.sum(X[:, 1] == 1)\n",
    "    p_hat_z1_y1 = np.sum((X[:, 2] == 1) & (X[:, 1] == 1)) / np.sum(X[:, 1] == 1)\n",
    "    p_hat_z2_y1 = np.sum((X[:, 2] == 2) & (X[:, 1] == 1)) / np.sum(X[:, 1] == 1)\n",
    "    \n",
    "    theta_hat = np.array([\n",
    "        p_hat_x0 * p_hat_y0_x0 * p_hat_z0_y0,\n",
    "        p_hat_x0 * p_hat_y0_x0 * p_hat_z1_y0,\n",
    "        p_hat_x0 * p_hat_y0_x0 * p_hat_z2_y0,\n",
    "        \n",
    "        p_hat_x0 * p_hat_y1_x0 * p_hat_z0_y1,\n",
    "        p_hat_x0 * p_hat_y1_x0 * p_hat_z1_y1,\n",
    "        p_hat_x0 * p_hat_y1_x0 * p_hat_z2_y1,\n",
    "        \n",
    "        p_hat_x1 * p_hat_y0_x1 * p_hat_z0_y0,\n",
    "        p_hat_x1 * p_hat_y0_x1 * p_hat_z1_y0,\n",
    "        p_hat_x1 * p_hat_y0_x1 * p_hat_z2_y0,\n",
    "        \n",
    "        p_hat_x1 * p_hat_y1_x1 * p_hat_z0_y1,\n",
    "        p_hat_x1 * p_hat_y1_x1 * p_hat_z1_y1,\n",
    "        p_hat_x1 * p_hat_y1_x1 * p_hat_z2_y1\n",
    "    ])\n",
    "    \n",
    "    return theta_hat\n",
    "\n",
    "true_distribution = np.array([\n",
    "    0.5 * 0.75 * 0.5,\n",
    "    0.5 * 0.75 * 0.25,\n",
    "    0.5 * 0.75 * 0.25,\n",
    "\n",
    "    0.5 * 0.25 * 0.25,\n",
    "    0.5 * 0.25 * 0.5,\n",
    "    0.5 * 0.25 * 0.25,\n",
    "\n",
    "    0.5 * 0.25 * 0.5,\n",
    "    0.5 * 0.25 * 0.25,\n",
    "    0.5 * 0.25 * 0.25,\n",
    "\n",
    "    0.5 * 0.75 * 0.25,\n",
    "    0.5 * 0.75 * 0.5,\n",
    "    0.5 * 0.75 * 0.25\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 1000\n",
    "X = generate_samples(n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "theta_hat = estimate_parameters(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1QAAAI0CAYAAAAa4mizAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzde5yWdZ0//tdwEBUtwmYg7WjteozFQy2isj88MIoghFQqAUVirushK1HzlJrKtu6SWbmb22F1oTAzkFQgj48UrKB18ZtgS6KpGIzgCQVlmOv3xzyc25HDwMXADPB8Ph49Hlz3fd33/b5ffGSuV/d1X1NVFEURAAAANlmHth4AAABgW6VQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAbRD++yzTwYPHpwhQ4Y0+9+zzz67wceNGTMmy5cvT5KMHTs2CxcubJV55s2bl8suu2yTH3fllVfmhhtuaJUZNsczzzyTs88+e7OeY+TIkZk+fXorTbR1XXLJJfl//+//tfUYANulTm09AADr9l//9V/p3r37Jj3m4YcfbvrzTTfd1GqzLFy4MEuWLGm159vaFi9enEWLFrX1GG1m1qxZ+exnP9vWYwBslxQqgG3Ma6+9losuuihPP/10OnTokAMOOCBXXnllLr744iTJ6NGj84Mf/CAjRozI9ddfn9dffz3/9m//lve9731ZtGhRdtlll5x++um55ZZbsmjRogwYMCBf//rX09DQkGuuuSb/+7//m9deey1FUeSb3/xm9txzz3znO9/Jq6++mosuuijXXntt7rvvvtx4441ZvXp1dt5551xwwQU56KCDsmLFilx88cVZsGBBampq0rFjxxxyyCHN5l+zZk2OOuqofO9738uBBx6YJPnyl7+cT37yk/n7v//7XHzxxXnzzTdTFEWGDx+eESNGrJXBv//7v+fee+/NqlWrsnLlylxwwQU59thjU19fn3/5l3/JAw88kI4dO+aggw7K5ZdfnksuuSRLlizJF7/4xVxxxRUZPHhw/ud//idJ8uyzzzZtv/766/nGN76Rp59+Oi+99FK6du2a6667Lnvvvfd6/z5GjhyZ/fffP3Pnzs2LL76YIUOG5JxzztngnDfccEMeffTRLF26NPvss08uvPDCXHbZZVm2bFnq6uqy11575dvf/nb22GOPHHXUURk0aFAeeeSRvPzyyznttNPyhz/8IX/84x/TqVOn3HjjjenRo0eWLFmSK6+8Ms8//3xWr16dE044IWeccUYmTJiQpUuX5mtf+1q+9a1vZe+9987VV1+dP/3pT1m9enUOO+ywjBs3Lp06dcqBBx6Yo48+OgsWLMh1112X+++/P7/+9a/TuXPnvOc978m1116bmpqaVlnHANuNAoB252//9m+LQYMGFSeeeGLT/84888yiKIril7/8ZTFmzJiiKIqivr6+uPjii4unnnqq6XHLli0riqIo+vfvX8ybN6945JFHiv3226/44x//WBRFUXzxi18sPvvZzxZvvPFGsWzZsuKAAw4o/vrXvxZ/+MMfirPPPrtYs2ZNURRF8R//8R/Fl770paIoiuIXv/hFcfrppxdFURSLFi0qBg0aVCxfvrwoiqL405/+VBx++OHFa6+9Vlx99dXFuHHjioaGhmLZsmVFv379iu985ztrvb/rr7++uOKKK4qiKIqXXnqp+OQnP1m88sorxUUXXVT8x3/8R1EURbF06dLiy1/+ctM8b3n22WeLkSNHFitXriyKoih+9atfFYMGDSqKoij+67/+qxgxYkSxcuXKYs2aNcW5555b/PKXvyweeeSR4oQTTiiKoiieeeaZonfv3k3P9/btu+++u7jqqqua7rv00kuLK6+8siiKovjc5z5X3H333Wu9l8997nPF2LFjizfffLN4+eWXi9ra2uK+++7b4Jzf+c53itra2mL16tVFURTFT37yk6b33dDQUJx22mnFD3/4w6a/x2uuuaYoiqK48847i3333beYP39+URRFceaZZxY33nhjURRFMXLkyOLee+8tiqIoVq1aVYwcObK48847m62FoiiKCy+8sLj55puLomhcP1/72teKH/zgB0VRNK6fX/7yl0VRFMXixYuLgw8+uHjjjTeKoiiKH/7wh8Wvf/3rtd4/wI7OJ1QA7dT6Tvk75JBDMmHChIwcOTJ9+/bN6NGj86EPfWiDz/X+978/+++/f5Lkgx/8YHbffffstNNO6d69e7p27ZqXX345Bx10UN797nfnZz/7WZ555pn89re/TdeuXdd6rocffjhLly7N5z//+abbqqqq8pe//CWzZ8/O17/+9VRVVaV79+459thj1znPSSedlOHDh+fCCy/Mr371qxx11FHZfffdc+yxx+aCCy7IvHnzcthhh+WSSy5Jhw7Nv+6711575Vvf+lamTZuWp59+uukTtaTx1LYhQ4Zk5513TpJ8+9vfTpL89re/3WA+bznuuOPygQ98ILfcckuefvrp/O53v8tBBx3U4uM++9nPpnPnzuncuXOOO+64PPTQQ+nfv/9650yS3r17p1Onxh/Do0ePzpw5c/LjH/84Tz31VP7v//4vf/d3f9e074ABA5IkH/jAB/Le9743++67b5LGv8uXX345r7/+en7/+9/n5ZdfzvXXX58kef3117NgwYIMHDiw2awPPPBAHnvssdx2221JklWrVjW7/9BDD02S9OjRI/vuu28+9alPpV+/funXr18OO+ywjcoRYEeiUAFsYz7wgQ/k17/+dX7729/mkUceyRe+8IVceeWVOeqoo9b7mJ122qnZ9lsH8m/3wAMP5Oqrr84XvvCFHH300dl7771zxx13rLVfQ0NDDjvssKaykiTPP/9806lgRVE03d6xY8d1zrPXXntl//33zwMPPJDbb789X//615Mk/fv3z4wZMzJr1qzMnj073/ve93L77benZ8+eTY/94x//mDPPPDOf//znc/jhh+cTn/hErrjiinW+rxdeeCENDQ3Nbquqqmo24+rVq5v+PGnSpNx6660ZMWJEBg8enG7durV4IZB3vm5RFOnQocMG50ySXXfdtenP//Iv/5J58+blpJNOyt///d+nvr6+2Yxv//vr3LnzWq/f0NCQoijys5/9LLvsskuSZPny5enSpcs6973++uvz0Y9+NEnyyiuvpKqqaq25OnTokP/+7//OY489ltmzZ+eaa67JkUcemXHjxrWYB8COxFX+ALYxkyZNykUXXZQjjjgi559/fo444og8/vjjSRoLTH19fannffjhh9O/f/+ceuqpOfDAA3PPPfdkzZo1az3vYYcdlocffjh//vOfkyQPPvhgTjzxxKxatSpHHnlkbrvttjQ0NOTll1/Ovffeu97X+8xnPpObbropK1eubPqe1Ve/+tXcddddOeGEE3L55Zdnt912y1/+8pdmj/v973+fAw88MF/4whfyyU9+Mvfee2/TnIcddlh+9atf5c0330xDQ0O+8Y1v5M4770zHjh2bitO73vWurF69uukKiHfeeWfTcz/00EP51Kc+lU9/+tP5yEc+kvvuu6/puTfkjjvuaHrPd999d4466qgNzvlODz30UEaPHp2hQ4dmjz32yKxZszbqdd+y2267pXfv3vnxj3+cpLEknXLKKU35v/3v74gjjshPfvKTFEWRN998M//4j/+Y//7v/17rORcsWJBBgwblox/9aL70pS/l85//fB577LGNnglgR+ETKoB2avTo0Wud7vaVr3wlQ4cOze9+97sMHDgwu+yyS973vvdl5MiRSRpPWRs5cmSpS5WffPLJ+epXv5rBgwenvr4+hx9+eGbOnJmGhob07t073/ve93LWWWflu9/9bq688sp85StfSVEUTRdG6Nq1a84+++xcfvnlOf7449O9e/f87d/+7Xpf76ijjsoVV1yRsWPHNt125pln5uKLL87kyZPTsWPHHHPMMfnEJz7R7HGDBg3KzJkzc/zxx6ehoSH9+/fPyy+/nBUrVuTkk0/Oc889l2HDhqUoinzyk5/MyJEjs2LFinTp0iXDhw/Pz3/+85x//vkZO3ZsunfvnuOOO67puceMGZPLLrus6XS43r17509/+lOL2a1atSrDhw/Pa6+9llNPPTWHHXZY/uZv/ma9c77TP/3TP+Vb3/pWrr/++nTu3DkHH3zwWkWyJdddd12uuuqqDB48OG+++WYGDRqUE088MUly7LHH5vzzz883vvGNXHzxxbn66qszePDgrF69On379s1pp5221vPtu+++Of7443PSSSdl1113zc4775xLLrlkk2YC2BFUFW8/pwAA2CQjR47MiBEjmhUzAHYcTvkDAAAoySdUAAAAJfmECgAAoCSFCgAAoCSFCgAAoCSFCgAAoKR2/XuoXnzxtTQ0tJ9rZuyxx25Ztmzt3x+yI5JFIzlUyKJCFo3kUCGLClk0kkOFLCpkUdGesujQoSrveU/X9d7frgtVQ0PRrgpVknY3T1uSRSM5VMiiQhaN5FAhiwpZNJJDhSwqZFGxrWThlD8AAICSFCoAAICS2vUpfwAAQMWaNfV58cW61Ne/2dajbFFLl3ZIQ0PDVn/dTp12ynveU52OHTe+JilUAACwjXjxxbrsvPOu6dq1Z6qqqtp6nC2mU6cOqa/fuoWqKIq89torefHFurz3ve/b6Mc55Q8AALYR9fVvpmvXd23XZaqtVFVVpWvXd23yp38KFQAAbEOUqS2nTLYKFQAA7CCm/ObJth5hu+M7VAAAsIO44+GnMvTIvVvt+Z5/fnFOOWVYPvzh5s85ePDQnHTSZ9ba/+GHf5Nnnnk6J5/8uUyZcluSZOjQ4Zs1w9ufc2OdddbpGTPm9Bx88KGb9dqJQgUAAGyG9763Oj/5yaSN2nfBgseb/ry5RWpdz9kWFCoAAKBV1dfX59prr8iTT/45SfKpT306H//432Xq1NuTJD17vi9//evzSZIvfvFLOfHE2hx55D/k8cf/X7p3f29OPHFIJk/+aerqlubrX788Bx10SP7nf+bmBz/4ft54Y1VefXVFzjnnvLz//R9s9pz9+x+Tf/u3f86TT/45DQ0NGTFiVI499ri8+eab+ed/vioLFsxPz5575uWXX2q196pQAQAApb3wQl0+//lTm9127rlfzSuvvJIf/3hSXnihLjfeeENOPPFTGTJkWJLkhBNOzA9/+B9N+y9fvix9+vTN+ed/PWef/aU8+OD9+f73/zN33/2r3HrrT3PQQYfkF7+YnAsvvDQf+tCHM3fu73P99dfl5psnN3vOG2+8Ifvss18uueSKvPbaipxxxpjsv/+BefDB+5MkEyfelmee+UtGjz6l1d6/QgUAANuhKb95Mnc8/NRat48Zf1+z7RMP//Bmfa9qXaf8vfzyS/nLX57OV75yVvr0OTz/9E/ntvg8ffocnqTxk6bevQ9KkvTo0TOvvvpKkuTSS6/KrFm/yf3335M//vGxrFy5cq3nmDPnd3njjVW58847kiSrVq3KokVP5tFH5+bEExuL1wc+8MF8/OO9Sr/fd1KoAABgOzT0yL3XKkpjxt+XH1141BZ/7Xe/u1tuueXW/P73v83s2Q9nzJjP5ZZbbt3gYzp37tz0544dO651/z/909gcfPAhOeigQ3LIIZ/IFVdcstY+DQ1rcumlV2WfffZN0vjJ17ve9e7ccccvkxQbfP6yXDYdAABoVQ899GCuuuqy9O17RL785a9ll112ydKlS9KxY8esWbNmk5/vlVdezjPPPJ0vfvGM9OlzeH7zmwfT0NCQJM2e8+CDP9F09cAXXngho0efkiVL/ppDD/1kZs6cnoaGhvz1r8/nscfmtdp79QkVAABQ2rq+Q3XggR9Ply5dMnLkZ7LTTjultnZgPvrRj+XVV1/J1Vd/I927d9+k13jXu96dQYOGZOTIz6RTp045+OBPZNWqVVm5cmV69z646TnHjBmbf/3Xf87IkZ9JQ0NDzjzznOy11/szbNins2jRnzNixPD07Pm+7L33R1vt/VcVRVG0vFvbWLZsRRoa2ma8Nx69Kx1rPpJOe+7XdFt19e6pq3u1abt+8fysWbooXXoPbIsR29Q7s9hRyaFCFhWyaCSHCllUyKKRHCp21CzKHmv+9a9Pp2fPD5V+3a11yt/m6tSpQ+rrG9rktd+ZcYcOVdljj93Wu79T/tajY81Hsuqe76d+8fx13l+/eH5W3fP9dKz5yFaeDACAbV1bHWueePiHW/X5UKjWq9Oe+2XnY85c50J/a4HvfMyZzf5fBQAA2Bhtday5OVfzY90Uqg1Y10JXpgAAaA2ONbcPLkrRgrcv9OUv/Tmr5sywwAEAaBWONbd9PqHaCJ323C+d9++flx66LZ3372+BAwDQahxrbtsUqo1Qv3h+Vj9+f7odMTyrH79/vV8eBACATbWljjXfePSuFp+rfvH8vPHoXa3yejsqhaoFbz+Ptfs/nLLeLw8CAMCm2pLHmq5avXUoVBuwri8FbuiKLAAAsLG29LHm1ryS4JNPLswRRxyaBx64d4P7LV78XK699srSr3PEEYeWfuyWolCtx4YWmVIFAMDm2FrHmlvrSoJ33nlH+vc/JlOn3r7B/f761+fz3HPPtsprthcK1XqsWbpog4vsrcW5ZumirTwZAADbuq15rPn2UvXGnNtbvUzV19dn5szpGTv2H/OnPy1oKky///1vM3r0KRk16rMZN+7Lee21Fbn++uvyxBPz86//+s/5wx/m5KyzTm96nquv/kbuumtakuTGG7+b00//fE4+eVjOOuv0LF++rFVm3RIUqvXo0ntgi4us0577pUvvgVtpIgAAthdb+1jzrSsJvvmHO1r9SoKzZj2Unj175oMf/FCOPPL/y9Spt+fNN9/MlVdemksu+UZuvnly9t77Y7n77l/l3HO/ln322S9f/eoF632+Z599Jk8//VT+/d9/lJ/97Pb06NEzM2bc3WrztjaFCgAAtnNvXUlwp4NPbPWrVt911x055pjaJMnRRx+bu+6aloUL/5Tq6ur8zd/skyQ544yzMnz4yRv1fO9//wdyzjlfybRpU3LDDRPyxz8+lpUrX2+1eVubX+wLAADbsXd+Z6rjnvu12ml/L764PI88MitPPLEgP//5z1IURV599ZU88sisJFVN+61YsSKvv/5as8dWVVU1266vr0+SLFgwP1dccXE++9lT07//0enYsUOKotisObckn1ABAMB2aktfSXD69LtyyCGfzC9/eVduu21afvGLX2XUqDF55JFZeemlF7No0ZNJkokT/ytTpvwiHTt2ypo1a5Ik7353tyxe/FzeeOONvPLKy/nf//2fJMmjj87NwQcfkqFDh+cDH/hgZs16KA0NDZs155akUAEAwHZoa1xJ8O67p+VTnxre7LZhwz6TP//5/3LppVfmm9+8PKNHn5ynnlqUz33u8/nwhz+cFStezVVXXZq99/5oDjvs8Iwc+ZlceumF+bu/OyhJcvTRA/J///d/GTXqszn77C9ln332y/PPLy4945ZWVbTjz8+WLVuRhob2M96v//Bcjj14r7Yeo12ort49dXWvtvUYbU4OFbKokEUjOVTIokIWjeRQIYuKjcnir399Oj17fmijnu+NR+9Kx5qPbPC0vvrF87Nm6aJ2d6G1Tp06pL6+bT6VemfGHTpUZY89dlvv/j6h2gQ/nflEW48AAAAbxVWrtw6FCgAAoCSFCgAAtiHt+Bs727wy2SpUAACwjejQoWPWrKlv6zG2W2vW1KdDh46b9Bi/h2o9pvzmydzx8FNr3T5m/H3Ntk88/MMZeuTeW2kqAAB2ZLvsslteffWldOu2R6qqfDbSmoqiIa+++mJ22WX9F6BYF4VqPYYeufdaRWnM+PvyowuPaqOJAADY0e2227vz4ot1WbLk2STb76l/HTp0aIPfPVWVnXbaObvt9u5NepRCBQAA24iqqqp0717T1mNscdvS5fQ36nPCadOmZeDAgRkwYEAmTpy43v3GjRuX22+/vWn72WefzYgRIzJkyJCMHDkyzz333OZPDAAA0E60WKiWLFmSCRMmZNKkSZkyZUomT56chQsXrrXPGWeckRkzZjS7/frrr88JJ5yQqVOnZsCAAZkwYULrTg8AANuJSTMWtPUIlNBioZo1a1b69OmTbt26Zdddd01tbW2mT5/ebJ9p06bl6KOPzvHHH9/s9oaGhqxYsSJJsnLlyuy8886tOPrWd8qAfdp6BAAAtlM/nflEW49ACS1+h2rp0qWprq5u2q6pqcm8efOa7XPaaaclSebOndvs9nPPPTcnn3xybrnllqxevTqTJ09ujZnbzKm1+24z53ICAABbXouFqqGhIVVVVU3bRVE0296QCy64IFdeeWWOOeaYzJgxI2eddVbuuOOOjX78Hnts2iULt4bq6t3beoR2QxaN5FAhiwpZNJJDhSwqZNFIDhWyqJBFxbaSRYuFqmfPnpkzZ07Tdl1dXWpqWr6yyPLly/Pkk0/mmGOOSZLU1tbm8ssvz4svvpju3btv1HDLlq1IQ0P7uRzktnS1kS1NFo3kUCGLClk0kkOFLCpk0UgOFbJoThaN2tO66NChaoMf9LRYqPr27Zsbbrghy5cvzy677JKZM2fmqquuavGF3/Oe96RLly6ZM2dODj300MydOzddu3bd6DIFAADbqym/eTJ3PPzUWrePGX9fs+0TD//wWr8blfalxULVo0ePnHfeeRk1alRWr16d4cOHp1evXhk7dmzOOeecfPzjH1/n46qqqvLd7343V111VVatWpWuXbvmhhtuaPU3AAAA25qhR+69VlEaM/6+/OjCo9poIsraqF/sO3jw4AwePLjZbTfddNNa+40fP77Zdq9evfLzn/98M8YDAABovzbqF/sCAACwNoUKAACgJIUKAADagVMG7NPWI1CCQgUAAO3AqbX7tvUIlKBQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlKRQAQAAlLRRhWratGkZOHBgBgwYkIkTJ653v3HjxuX2229v2l66dGlOP/30DB06NCeffHKeffbZzZ8YAACgnWixUC1ZsiQTJkzIpEmTMmXKlEyePDkLFy5ca58zzjgjM2bMaHb7uHHj0r9//0yZMiVDhgzJdddd17rTAwAAtKFOLe0wa9as9OnTJ926dUuS1NbWZvr06TnrrLOa9pk2bVqOPvropn2SZPny5VmwYEF+/OMfJ0lOOumkHHbYYa09PwAAQJtpsVAtXbo01dXVTds1NTWZN29es31OO+20JMncuXObbnvmmWey5557Zvz48ZkzZ06qq6tz6aWXttbcAAAAba7FQtXQ0JCqqqqm7aIomm2vT319fR5//PGcffbZueiii/Lzn/88F154YW655ZaNHm6PPXbb6H23lurq3dt6hHZDFo3kUCGLClk0kkOFLCpk0UgOFbKokEXFtpJFi4WqZ8+emTNnTtN2XV1dampqWnzi6urqdO3aNf3790+SDBo0KN/85jc3abhly1akoaHYpMdsSdXVu6eu7tW2HqNdkEUjOVTIokIWjeRQIYsKWTSSQ4UsKmRR0Z6y6NChaoMf9LR4UYq+fftm9uzZWb58eVauXJmZM2emX79+Lb7wBz/4wfTs2TMPPvhgkuT+++/PAQccsAmjAwAAtG8tFqoePXrkvPPOy6hRozJ06NAMGjQovXr1ytixY/PYY49t8LE33HBD/vM//zODBg3KzTffnGuuuabVBgcAAGhrVUVRtJ9z6t7BKX/tlywayaFCFhWyaCSHCllUyKKRHCpkUSGLivaUxWaf8gcAAMC6KVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlKVQAAAAlbVShmjZtWgYOHJgBAwZk4sSJ691v3Lhxuf3229e6/fHHH8+BBx5YfkoAAIB2qMVCtWTJkkyYMCGTJk3KlClTMnny5CxcuHCtfc4444zMmDFjrcevXLkyV111VVavXt16UwMAALQDLRaqWbNmpU+fPunWrVt23XXX1NbWZvr06c32mTZtWo4++ugcf/zxaz1+/PjxGT16dOtNDAAA0E50ammHpUuXprq6umm7pqYm8+bNa7bPaaedliSZO3dus9vvvfferFq1Kscdd1xrzAoAANCutFioGhoaUlVV1bRdFEWz7fWpq6vLjTfemJ/85Celh9tjj91KP3ZLqa7eva1HaDdk0UgOFbKokEUjOVTIokIWjeRQIYsKWVRsK1m0WKh69uyZOXPmNG3X1dWlpqamxSd+4IEH8tJLL2XEiBFNtw0ZMiQTJ07MbrttXFFatmxFGhqKjdp3a6iu3j11da+29RjtgiwayaFCFhWyaCSHCllUyKKRHCpkUSGLivaURYcOVRv8oKfF71D17ds3s2fPzvLly7Ny5crMnDkz/fr1a/GFP/3pT+eee+7J1KlTM3Xq1CTJ1KlTNyticHIAAB9uSURBVLpMAQAAtHctFqoePXrkvPPOy6hRozJ06NAMGjQovXr1ytixY/PYY49tjRkBAADapRZP+UuSwYMHZ/Dgwc1uu+mmm9bab/z48et9jieeeGITRwMAAGjfNuoX+wIAALA2hQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKAkhQoAAKCkjSpU06ZNy8CBAzNgwIBMnDhxvfuNGzcut99+e9P23LlzM3z48AwZMiSjR4/Oc889t/kTAwAAtBMtFqolS5ZkwoQJmTRpUqZMmZLJkydn4cKFa+1zxhlnZMaMGc1uP//88/PNb34zU6dOzeDBg/PNb36zdacHAABoQy0WqlmzZqVPnz7p1q1bdt1119TW1mb69OnN9pk2bVqOPvroHH/88U23vfnmmzn33HOz7777Jkn22WefPP/88608PgAAQNvp1NIOS5cuTXV1ddN2TU1N5s2b12yf0047LUnjKX5v2WmnnTJkyJAkSUNDQ7773e/mmGOO2aTh9thjt03af2uort69rUdoN2TRSA4VsqiQRSM5VMiiQhaN5FAhiwpZVGwrWbRYqBoaGlJVVdW0XRRFs+2WvPnmm7nwwgtTX1+fL33pS5s03LJlK9LQUGzSY7ak6urdU1f3aluP0S7IopEcKmRRIYtGcqiQRYUsGsmhQhYVsqhoT1l06FC1wQ96Wjzlr2fPnqmrq2varqurS01NzUa9+GuvvZbTTjst9fX1ufHGG9O5c+eNehwAAMC2oMVC1bdv38yePTvLly/PypUrM3PmzPTr12+jnvz888/Phz70oXz729/OTjvttNnDAgAAtCctnvLXo0ePnHfeeRk1alRWr16d4cOHp1evXhk7dmzOOeecfPzjH1/n4x5//PHce++9+djHPpZPfepTSRq/f3XTTTe17jsAAABoIy0WqiQZPHhwBg8e3Oy2dRWj8ePHN/15//33zxNPPLGZ4wEAALRfG/WLfQEAAFibQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFCSQgUAAFDSRhWqadOmZeDAgRkwYEAmTpy43v3GjRuX22+/vWl78eLFGTFiRI477rj84z/+Y1577bXNnxgAAKCdaLFQLVmyJBMmTMikSZMyZcqUTJ48OQsXLlxrnzPOOCMzZsxodvsVV1yRU089NdOnT8+BBx6Y73//+607PQAAQBtqsVDNmjUrffr0Sbdu3bLrrrumtrY206dPb7bPtGnTcvTRR+f4449vum316tX5/e9/n9ra2iTJsGHD1nocAADAtqxTSzssXbo01dXVTds1NTWZN29es31OO+20JMncuXObbnvxxRez2267pVOnxpeorq7OkiVLNmm4PfbYbZP23xqqq3dv6xHaDVk0kkOFLCpk0UgOFbKokEUjOVTIokIWFdtKFi0WqoaGhlRVVTVtF0XRbHt91rXfxjzu7ZYtW5GGhmKTHrMlVVfvnrq6V9t6jHZBFo3kUCGLClk0kkOFLCpk0UgOFbKokEVFe8qiQ4eqDX7Q0+Ipfz179kxdXV3Tdl1dXWpqalp84e7du+fVV1/NmjVrNulxAAAA24oWC1Xfvn0ze/bsLF++PCtXrszMmTPTr1+/Fp+4c+fOOfTQQ3PXXXclSaZMmbJRjwMAANhWtFioevTokfPOOy+jRo3K0KFDM2jQoPTq1Stjx47NY489tsHHXn755bn11lszcODAzJkzJ1/+8pdbbXAAAIC2VlUURfv5ktI7+A5V+yWLRnKokEWFLBrJoUIWFbJoJIcKWVTIoqI9ZbHZ36ECAADYmibNWNDWI2w0hYpStqVFDgDAtuWnM59o6xE2mkJFKdvSIgcAgC1FoQIAAChJoQIAACipU1sPAAAA7Lim/ObJ3PHwU2vdPmb8fc22Tzz8wxl65N5baaqNp1DRom19kQMA0H4NPXLvtY4hx4y/Lz+68Kg2mmjTKFS0aFtf5AAAsKX4DhUteuPRu1K/eP4G96lfPD9vPHrXVpqobcgBADaPn6Wsy7a+LhQqWtSx5iNZdc/317vQ6xfPz6p7vp+ONR/ZypNtXXIAgM3jZynrsq2vC4WKFnXac7/sfMyZ61zoby3wnY85M5323K+NJtw65AAAm8fPUtZlXevilAH7JNk21oVCxUZ550I/ZcA+28QCb23r+g9+R8wBAMrys5R1eee6OLV2321mXbgoBRvt7Qv9uENr8/I9M9r9At8S3p7D8pf+nFVzdswcAKAsP0tZl211XfiEik3Sac/90nn//nnpodvSef/+7X6BbylyAIDN42cp67ItrguFik1Sv3h+Vj9+f7odMTyrH7+/xSuybK/kAACb562fpdNX9vKzlCbb4jGWQsVGe/t5rN3/4ZT1fql0eycHANg8b/9ZevfK3n6WkmTbPcZSqNgo6/pS4Iau1LO9kgMAbB4/S1mXbXldKFS0aENXWNlWFnprkAMAbB4/S1mXbX1duMofLVqzdNEGr7Dy1kJfs3TRNvHFwbLkAACbZ97v5mT60sOy8ObnkzzfdPuY8fc1/fljnQ7Lcb+bk4OH+lm6o9jWj7EUKlrUpffAFvfptOd+7XKBtyY5AMDmOXjoyBz8jtvGjL8vP7rwqDaZh/ZhWz/GcsofAABASQoVAABASQoVAABASQoVbIZJMxa09QgAsE07ZcA+bT0CbBaFCjbDT2c+0dYjAMA27dTafdt6BNgsChUAAEBJChUAAEBJChUAAEBJfrEvbKQpv3kydzz81Fq3v/23uyfJiYd/OEOP3HsrTQUAQFtSqGAjDT1y77WKkt/uDgCwY3PKHwAAQEkKFQAAQEkKFQAAQEkKFWwGv90dAGDHplDBZvDb3QEAdmwKFQAAQEkKFQAAQEkKFQAAQEkKFQAAQEkKFQAAQEkKFQAAQEkKFQAAQEkKFQAAQEkKFQAAQEkKFQAAQEkKFQAAQEkKFQAAQEkKFQAAQEkKFQAAQEkKFQAAQEkKFQAAQEkbVaimTZuWgQMHZsCAAZk4ceJa98+fPz/Dhg1LbW1tLr744tTX1ydJnn322YwYMSJDhgzJyJEj89xzz7Xu9AAAAG2oxUK1ZMmSTJgwIZMmTcqUKVMyefLkLFy4sNk+559/fi677LLMmDEjRVHk1ltvTZJcf/31OeGEEzJ16tQMGDAgEyZM2DLvAgAAoA20WKhmzZqVPn36pFu3btl1111TW1ub6dOnN93/3HPPZdWqVendu3eSZNiwYU33NzQ0ZMWKFUmSlStXZuedd94S7wEAAKBNdGpph6VLl6a6urppu6amJvPmzVvv/dXV1VmyZEmS5Nxzz83JJ5+cW265JatXr87kyZM3abg99thtk/bfGqqrd2/rEdoNWTSSQ4UsKmTRSA4VsqiQRSM5VMiiQhYV20oWLRaqhoaGVFVVNW0XRdFse0P3X3DBBbnyyitzzDHHZMaMGTnrrLNyxx13NNt/Q5YtW5GGhmKj38yWVl29e+rqXm3rMdoFWTSSQ4UsKmTRSA4VsqiQRSM5VMiiQhYV7SmLDh2qNvhBT4un/PXs2TN1dXVN23V1dampqVnv/S+88EJqamqyfPnyPPnkkznmmGOSJLW1tamrq8uLL75Y6o0AAAC0Ny0Wqr59+2b27NlZvnx5Vq5cmZkzZ6Zfv35N9++1117p0qVL5s6dmySZOnVq+vXrl/e85z3p0qVL5syZkySZO3duunbtmu7du2+htwIAALB1tXjKX48ePXLeeedl1KhRWb16dYYPH55evXpl7NixOeecc/Lxj3881113XS655JKsWLEiBxxwQEaNGpWqqqp897vfzVVXXZVVq1ala9euueGGG7bGewIAANgqqoqiaD9fUnoH36Fqv2TRSA4VsqiQRSM5VMiiQhaN5FAhiwpZVLSnLDb7O1QAAACsm0IFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIFAABQkkIF0MomzVjQ1iMAAFuJQgXQyn4684m2HgEA2EoUKgAAgJIUKgAAgJIUKgDYwnyvDmD71amtBwDYlk35zZO54+Gn1rp9zPj7mm2fePiHM/TIvbfSVLQ3P535RI49eK+2HgOALUChAtgMQ4/ce62iNGb8ffnRhUe10UQAwNbklD8AAICSFCoAAICSnPIHAK3I9+oAdiwKFcBmeOPRu9Kx5iPptOd+TbedMmCfZvvUL56fNUsXpUvvgVt7PNrA8bsvyKBRzdfEO79X17gmFiRRqAC2dU75A9gMHWs+klX3fD/1i+c33XZq7b5Nf65fPD+r7vl+OtZ8pC3Gow2sa028nTUBsH1RqAA2Q6c998vOx5y5zgPotw6cdz7mzGafVrB9syYAdiwKFcBmWtcBtAPnHZs1AbDj8B0qgFbw9gPo5S/9OavmzHDgvIN7+5o4/4BDs+qeX1oTANshn1ABtJJOe+6Xzvv3z0sP3ZbO+/d34EzTmnj/8/dbEwDbKYUKoJXUL56f1Y/fn25HDM/qx+9f70UJ2HFYEwDbP4UKoBW8/fsx3f/hlPVelIAdhzUBsGNQqAA207ouNrChK72x/bMmAHYcG1Wopk2bloEDB2bAgAGZOHHiWvfPnz8/w4YNS21tbS6++OLU19cnSZYuXZrTTz89Q4cOzcknn5xnn322dacHaGMbunKbA+gdkzUBsGNpsVAtWbIkEyZMyKRJkzJlypRMnjw5CxcubLbP+eefn8suuywzZsxIURS59dZbkyTjxo1L//79M2XKlAwZMiTXXXfdlnkXAG1kzdJFG7xy21sH0GuWLtrKk9FWrAmAHUuLhWrWrFnp06dPunXrll133TW1tbWZPn160/3PPfdcVq1ald69eydJhg0blunTp2f58uVZsGBBTj755CTJSSedlC9/+ctb6G0AtI0uvQe2eOW2Tnvuly69B26liWhr1gTAjqXF30O1dOnSVFdXN23X1NRk3rx5672/uro6S5YsyTPPPJM999wz48ePz5w5c1JdXZ1LL710k4bbY4/dNmn/raG6eve2HqHdkEUjOVTIokIWjeRQIYsKWTSSQ4UsKmRRsa1k0WKhamhoSFVVVdN2URTNttd3f319fR5//PGcffbZueiii/Lzn/88F154YW655ZaNHm7ZshVpaCg2ev8trbp699TVvdrWY7QLsmgkhwpZVMiikRwqZFEhi0ZyqJBFhSwq2lMWHTpUbfCDnhZP+evZs2fq6uqatuvq6lJTU7Pe+1944YXU1NSkuro6Xbt2Tf/+/ZMkgwYNavbJFgAAwLauxULVt2/fzJ49O8uXL8/KlSszc+bM9OvXr+n+vfbaK126dMncuXOTJFOnTk2/fv3ywQ9+MD179syDDz6YJLn//vtzwAEHbKG3AQAAsPW1WKh69OiR8847L6NGjcrQoUMzaNCg9OrVK2PHjs1jjz2WJLnuuuty7bXX5rjjjsvrr7+eUaNGJUluuOGG/Od//mcGDRqUm2++Oddcc82WfTcAAABbUVVRFO3nS0rv4DtU7ZcsGsmhQhYVsmgkhwpZVMiikRwqZFEhi4r2lMVmf4cKAACAdVOoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAASlKoAAAAStqoQjVt2rQMHDgwAwYMyMSJE9e6f/78+Rk2bFhqa2tz8cUXp76+vtn9jz/+eA488MDWmRgAAKCdaLFQLVmyJBMmTMikSZMyZcqUTJ48OQsXLmy2z/nnn5/LLrssM2bMSFEUufXWW5vuW7lyZa666qqsXr269acHAABoQy0WqlmzZqVPnz7p1q1bdt1119TW1mb69OlN9z/33HNZtWpVevfunSQZNmxYs/vHjx+f0aNHb4HRAQAA2laLhWrp0qWprq5u2q6pqcmSJUvWe391dXXT/ffee29WrVqV4447rjVnBgAAaBc6tbRDQ0NDqqqqmraLomi2vb776+rqcuONN+YnP/lJ6eH22GO30o/dUqqrd2/rEdoNWTSSQ4UsKmTRSA4VsqiQRSM5VMiiQhYV20oWLRaqnj17Zs6cOU3bdXV1qampaXZ/XV1d0/YLL7yQmpqaPPDAA3nppZcyYsSIpvuGDBmSiRMnZrfdNq4oLVu2Ig0NxUbtuzVUV++eurpX23qMdkEWjeRQIYsKWTSSQ4UsKmTRSA4VsqiQRUV7yqJDh6oNftDT4il/ffv2zezZs7N8+fKsXLkyM2fOTL9+/Zru32uvvdKlS5fMnTs3STJ16tT069cvn/70p3PPPfdk6tSpmTp1atN9G1umAAAA2rsWC1WPHj1y3nnnZdSoURk6dGgGDRqUXr16ZezYsXnssceSJNddd12uvfbaHHfccXn99dczatSoLT44AABAW2vxlL8kGTx4cAYPHtzstptuuqnpz/vuu29uu+22DT7HE088UWI8AACA9mujfrEvAAAAa1OoAAAASlKoAAAASlKoANgiJs1Y0NYjAMAWp1ABsEX8dKaLEQGw/VOoAAAASlKoAAAASlKoAAAAStqoX+wL0JJJMxbk2IP3ausxaCNTfvNk7nj4qbVuHzP+vmbbJx7+4Qw9cu+tNBXtkX8rgO2NQgW0ip/OfMJB0g5s6JF7r1WUxoy/Lz+68Kg2moj2yr8VwPbGKX8AAAAlKVQAAAAlKVQAAAAl+Q4VsMlcgICNccqAfdp6BNqYfyuAHYFCBWwyFyBgY5xau2/q6l5t6zFoQ/6tAHYETvkDANjKJs1Y0NYjAK1EoQI22RuP3pX6xfM3uE/94vl549G7ttJEtDVrgnWxLhqtK4efznyi2faOkANsrxQqYJN1rPlIVt3z/fUeKNUvnp9V93w/HWs+spUno61YE6yLddFIDrB9U6iATdZpz/2y8zFnNjtAeOsCBG8dGOx8zJnptOd+bTkmW9G61sRbrIkdl38rGvnvA7ZvChVQyjsPEE6t3deBwQ5uXQeN1gT+rWjkvw/YfrnKH1Da2w8Qlr/056yaM8OBwQ7OmmBddvR18fbLx3+s02H5wh3fzvG77JO6Oybnxyv+IQtvfj7J8y4fD9sohQrYLJ323C+d9++flx66LTsdfOIOc4DE+lkTrMuOvC7eefn4N+bsnOP+cEd2OvjEfP3QYW04GdAanPIHbJb6xfOz+vH70+2I4Vn9+P0tXtGL7Z81wbpYF43eymH6yl47dA6wPVGogNLefv5/9384Zb1fumbHYU2wLtZFo7fncPfK3jtsDrC9UaiAUtb1ZeoNXcmK7Z81wbpYF43kANsvhQrYZBu6MpUDhB2TNcG6WBeN1pXDW5eP35FygO2VQgVssjVLF23wCl1vHSCsWbpoK09GW7EmWBfrotG6cji1dt+mP+8oOcD2ylX+gE3WpffAFvfptOd+O9RVvHZ01gTrYl00kgNs33xCBQAA/P/t201IlF0YxvFrQhBa1cIxiGhpYGmLoihQpNIsx0iKoIVBSEXQoo0QBi5CaBHUur2bqKCPxRgSrawWUbQoKCL6ohmNIjQ0pul+F+rYyzwzznMYZ/Q8/99uhmdxn8u5ONyjwhELFQAAAAA4YqECAAAAAEcsVAAAAADgiIUKAAAAAByxUAEAAACAIxYqAAAAAHDEQgUAAAAAjlioAAAAAMARCxUAAAAAOGKhAgAAAABHLFQAAAAA4IiFCgAAAAAcsVABAAAAgCMWKgAAAABwVFPtAYpZtSpW7RHyLMeZqoUsZpHDArJYQBazyGEBWSwgi1nksIAsFpDFguWSxWJzxMzMKjQLAAAAAHiFP/kDAAAAAEcsVAAAAADgiIUKAAAAAByxUAEAAACAIxYqAAAAAHDEQgUAAAAAjlioAAAAAMARCxUAAAAAOGKhAgAAAABHLFT/uHfvng4cOKD29nYNDw9LksbGxpRIJNTe3q6rV6/mnn39+rV6enrU0dGhgYEB/fnzp1pjL4kwWczr7+/X7du3Kz3qkguTxejoqA4dOqTu7m6dPXtWP3/+rNbYZReUgyRlMhmdOHFCT58+zb0XxX5IwVnMi1I/pOAsfO6HJE1NTamrq0ufP3+WFN37Qyo9i3m+9qPUHHzvhpSfhRTN+0MqPYt5vvZDKj2LZd8Rg5mZpVIpa2trsx8/ftivX78skUjYmzdvrLW11T5+/GiZTMZOnjxpjx49MjOzgwcP2vPnz83M7MKFCzY8PFzN8csqbBapVMpOnz5tTU1NduvWrSpPX15hspicnLTdu3dbKpUyM7Nr167ZpUuXqnyC8gjK4e3bt/bu3Ts7duyYbdmyxZ48eZJ7Pmr9KJZF1PpRKAuf+2Fm9uLFC+vq6rLGxkb79OmTTU9PR/L+MAuXhc/9KDUH37thlp+FmUXy/jALl4XP/TArPYuV0BF+QzVnbGxMO3fu1Jo1a7R69Wp1dHRoZGREGzdu1IYNG1RTU6NEIqFkMqkvX75oZmZGW7dulST19PQomUxW+QTlEyYLafYb6j179qizs7PKk5dfmCwymYwGBwdVX18vSWpoaNDXr1+rfILyCMohmUzq5s2b6uvrU3Nzc+7ZKPajUBZS9PpRKAuf+yFJN27c0ODgoOLxuCTp5cuXkbw/pNKzkPzuR6k5+N4NKT8LSZG8P6TSs5D87odUehYroSMsVHPGx8dVV1eXex2Px3X37t2899LpdN6zdXV1SqfTFZ13KYXJQpL6+vp09OjRis9ZCWGyWLt2rfbt2ydJmpmZ0fXr17V3796Kz7wUgnJIp9Pq7+/PO2MU+1EoCyl6/SiUhc/9kKShoSFt27Yt97pQNr73Qyo9C8nvfpSag+/dkPKzkBTJ+0MqPQvJ735IpWexEjrCQjXn79+/isViuddmJkl578ViscBn/3290oXJwncuWUxOTurUqVPatGmTDh8+XLlhl1CYz3wU++HT+cJwycLHfgQplE0UPz9RPHOQxXKISjeK4bOCYpZzR1io5qxbt04TExO51xMTE+ru7s57Lx6P5z377du3//26cqULk4XvwmYxPj6u48ePq6GhQUNDQxWfd6kE5VDo5x/Ffvh0vjDCZuFrP4IUysb3fgShM7OK5RClbhQTxX6gNMu9IyxUc3bt2qXHjx/r+/fvmp6e1oMHD7Rjxw69f/9eHz58UDab1f3799XS0qL169ertrZWz549kyTduXNHLS0tVT5B+YTJwndhsshmszpz5ow6Ozs1MDDg1bdqQTkU+vlHsR8+nS+MMFn43I8gzc3Nkbw/ghTKImoK5RC1bhQTxX5gcSuhIzXVHmC5qK+v1/nz59Xb26tMJqMjR45o+/btunz5ss6dO6ffv3+rtbVV+/fvlyRduXJFFy9e1NTUlBobG9Xb21vlE5RP2Cx8FiaL0dFRvXr1StlsViMjI5KkzZs3L8tvUsIKyqGpqang81HrR7EsfBYmi4cPH3rbjyC1tbWRvD+CFMsiSgrl4PPd4SJq/cDiVsL9EbP5fwoBAAAAAITCn/wBAAAAgCMWKgAAAABwxEIFAAAAAI5YqAAAAADAEQsVAAAAADhioQIAAAAARyxUAAAAAOCIhQoAAAAAHP0Hs+II8IJ0HO4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "\n",
    "sns.set()\n",
    "\n",
    "ind = np.arange(12)\n",
    "labels = ['000', '001', '002', '010', '011', '012', '100', '101', '102', '110', '111', '112']\n",
    "\n",
    "x = np.arange(len(labels))\n",
    "fig, ax = plt.subplots(figsize=(12, 8))\n",
    "ax.plot(x, theta_hat, label='Estimated', linestyle='', marker='+', ms=10)\n",
    "ax.plot(x, true_distribution, label='Actual', linestyle='', marker='x', ms=10)\n",
    "\n",
    "ax.set_title('Estimated vs actual parameters')\n",
    "ax.set_xticks(x)\n",
    "ax.set_xticklabels(labels)\n",
    "ax.legend()\n",
    "\n",
    "fig.tight_layout()\n",
    "\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "56c716d53c304cdfbc2e63febb8fd8de",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=10000), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# Bootstrapping:\n",
    "from tqdm import tqdm_notebook\n",
    "\n",
    "B = 10000\n",
    "parameters_bootstrap = np.empty((B, 12))\n",
    "for i in tqdm_notebook(range(B)):\n",
    "    XX = generate_samples(n)\n",
    "    parameters_bootstrap[i] = estimate_parameters(XX)\n",
    "    \n",
    "se_theta_hat = parameters_bootstrap.std(axis=0)\n",
    "theta_p05 = np.quantile(parameters_bootstrap, 0.05, axis=0)\n",
    "theta_p95 = np.quantile(parameters_bootstrap, 0.95, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1QAAAI0CAYAAAAa4mizAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeVhWdf7/8RebJsGMYqCto2bhQqZFgpq4iyiEmv7GJKQUl7FpsVFzqVzSLMdSK3NappmxtFWlKENFs1JxK01zqyZNJw1wCUFBlvvz++N8uRFBwSNygz4f19V1ec597nO/z5tP3OfF2dyMMUYAAAAAgAvm7uoCAAAAAKC6IlABAAAAgE0EKgAAAACwiUAFAAAAADYRqAAAAADAJgIVAAAAANhEoAJQ4QIDAxUVFaXo6Ohi//3vf/877/sGDx6sY8eOSZKGDh2qn376qULq2b59u55++ukLft/UqVP18ssvV0gNF+PgwYN6+OGHL2odsbGxSkpKqqCKKteTTz6p77///pKtPysrS0OGDFFOTk65lo+OjtaJEyeUmZmpQYMGOecHBgY6x29l2b17t7p27aq+fftqwYIFmjZtWqnLRUZGauPGjZVam13btm1TbGysoqKiFBkZqfj4eP3444/O1y/k90vnzp21Y8eOyixfkpSUlKTY2NjzLpOcnKx58+ZVUkUALiVPVxcA4PL0n//8R35+fhf0nnXr1jn//cYbb1RYLT/99JNSU1MrbH2V7dChQ9q3b5+ry3CZ9evX689//vMlW/+sWbPUv39/XXXVVeVa/uOPP5Yk/e9//3PJzvqZVq1apZCQEE2fPt2ldVSU3NxcDR8+XG+99ZaaN28uyer30KFDtWrVKnl4eEiy9/ulqunatasWLlyo3bt3q2nTpq4uB8BFIFABqFQnT57U+PHj9csvv8jd3V3NmzfX1KlTNXHiRElSXFycXn/9dcXExGju3Lk6deqUXnzxRV177bXat2+fatWqpWHDhuntt9/Wvn371L17d02YMEEOh0PPPvusvvvuO508eVLGGE2bNk3XXXedXnrpJWVmZmr8+PGaMWOGVq9erfnz5ysvL09XXXWVnnjiCbVq1UpZWVmaOHGi9uzZo4CAAHl4eOjOO+8sVn9BQYE6d+6sefPmKSgoSJL02GOPqXXr1goJCdHEiROVm5srY4z69eunmJiYEj34xz/+oVWrViknJ0fZ2dl64okn1K1bN+Xn5+vvf/+71qxZIw8PD7Vq1UqTJk3Sk08+qdTUVA0ZMkRTpkxRVFSUtm7dKsnaqS+cPnXqlCZPnqxffvlFv//+u66++mrNmjVLjRo1OufPIzY2Vs2aNdM333yj48ePKzo6Wo888sh563z55Ze1bds2paWlKTAwUOPGjdPTTz+to0ePKj09Xddff73mzJmjunXrqnPnzoqMjNSGDRuUkZGh+Ph4ffvtt9q5c6c8PT01f/581atXT6mpqZo6daoOHz6svLw89erVSyNGjNDs2bOVlpam0aNHa+bMmWrUqJGmT5+uH374QXl5eWrTpo3Gjh0rT09PBQUFqUuXLtqzZ49mzZqlL774QitXrpSXl5fq1KmjGTNmKCAgoNj2Hz58WF988YWefPJJ7d69WyNGjNCXX34pSRoyZIiuueYaPf/888rNzVX79u2VnJys4OBgpaSkaPz48crJyVF0dLSWLFkiSXr55Zf13Xff6ffff9eQIUNK/fl/9913mjZtmrKzs+Xl5aWxY8eqTZs22rJli2bOnOmc/9hjjyksLExLlizRypUr5e7url9++UVXXXWVnn/+ee3cuVPvvvuuCgoKlJOTo3bt2mn58uV67bXX9NNPP2nChAnKzs5Wo0aNdOrUKefnf/vtt5o1a5ays7Pl7u6uv/71r+rUqdM5P+fmm29Wenq6Jk2apJ9//lnu7u4aMGCABg0apMzMzHP+PEqzZMkSbdq0Sc8991ypr2dnZyszM7NYvffcc498fHxUUFDgDFQXYtGiRdqzZ49yc3P14IMPql+/fpKk999/X2+//bbc3d11zTXX6KmnnlLDhg01btw43XLLLRoyZIgkFZvu3Lmz+vTpo5SUFB0+fFjR0dF67LHHJElz585VYmKiateurT/96U/Oz9+yZYuee+45ORwOSdLw4cMVHh4uSerXr59eeeUVjlQB1Z0BgAp26623msjISHPPPfc4/xs5cqQxxpilS5eawYMHG2OMyc/PNxMnTjT79+93vu/o0aPGGGM6depktm/fbjZs2GCaNm1qdu7caYwxZsiQIebPf/6zOX36tDl69Khp3ry5+e2338y3335rHn74YVNQUGCMMea1114zw4cPN8YYs3jxYjNs2DBjjDH79u0zkZGR5tixY8YYY3744QfTrl07c/LkSTN9+nQzduxY43A4zNGjR01YWJh56aWXSmzf3LlzzZQpU4wxxvz++++mdevW5sSJE2b8+PHmtddeM8YYk5aWZh577DFnPYX+97//mdjYWJOdnW2MMebTTz81kZGRxhhj/vOf/5iYmBiTnZ1tCgoKzKOPPmqWLl1qNmzYYHr16mWMMebgwYOmZcuWzvWdOf3555+bZ555xvnaU089ZaZOnWqMMeb+++83n3/+eYltuf/++83QoUNNbm6uycjIMOHh4Wb16tXnrfOll14y4eHhJi8vzxhjzL///W/ndjscDhMfH2/++c9/On+Ozz77rDHGmM8++8w0adLE7N692xhjzMiRI838+fONMcbExsaaVatWGWOMycnJMbGxseazzz4rNhaMMWbcuHFmwYIFxhhr/IwePdq8/vrrxhhr/CxdutQYY8yhQ4fMHXfcYU6fPm2MMeaf//ynWblyZYntf/vtt80TTzzhnO7cubPZu3evyc7ONh07djRhYWHGGGPWrFlj4uPjnZ9z9OjREj+LW2+91bndO3fuNEFBQSY3N7fY5+Xm5pp27dqZL774whhjzI4dO5zjsU2bNmbbtm3GGGtctm7d2hw4cMAsXrzY3Hnnnebw4cPGGGOmTp1qxo4d6/xZFI7FM8d5dHS0+eCDD4wxxmzZssUEBgaaDRs2mN9//910797dHDx40BhjzG+//WbCwsLMr7/+et7Peeihh8zzzz9vjDHmxIkTplevXmb//v3n/XmUZvHixcX6XZq33nrLtGjRwnTu3NmMHj3afPjhh+bUqVPF+nyu3y9n69Spk5k0aZJzW9u0aWN++OEHs379etO1a1fn75vFixebiIgI43A4zBNPPGHefPNN5zrOnO7UqZN57rnnnOu77bbbzIEDB8zKlStNz549TWZmpsnLyzPDhg0z999/vzHGmEGDBplPP/3UGGPM7t27zeTJk53rPn78uAkKCnL+fwageuIIFYBL4lyn5Nx5552aPXu2YmNj1bZtW8XFxRX7a25pbrjhBjVr1kySdNNNN8nX11c1atSQn5+frr76amVkZKhVq1b64x//qPfee08HDx7Uxo0bdfXVV5dY17p165SWlqYHHnjAOc/NzU0HDhxQSkqKJkyYIDc3N/n5+albt26l1nPvvfeqX79+GjdunD799FN17txZvr6+6tatm5544glt375dbdq00ZNPPil39+KXql5//fWaOXOmEhMT9csvvziPqEnWqW3R0dHOU8/mzJkjSeW+9qVHjx668cYb9fbbb+uXX37Rpk2b1KpVqzLf9+c//1leXl7y8vJSjx49tHbtWnXq1OmcdUpSy5YtnUch4uLitGXLFv3rX//S/v379eOPP+r22293Ltu9e3dJ0o033qhrrrlGTZo0kWT9LDMyMnTq1Clt3rxZGRkZmjt3riTp1KlT2rNnj3r27Fms1jVr1mjHjh366KOPJKnEdU/BwcGSpHr16qlJkybq06ePwsLCFBYWpjZt2pTY9p9//lk33XSTc7pbt2766quvdMsttyg0NFR79+7Vjz/+qFWrVjm343wiIyMlSU2bNlVubq6ysrJUp04d5+s//PCD3N3d1bFjR0lSUFCQEhMT9eWXX+qmm25y9u2WW27RHXfcoU2bNsnNzU3NmzdX/fr1JUnNmjXTypUrz1nD8ePHtXfvXvXu3VuS9f/cLbfcIsm6Pik9PV0PPfSQc3k3Nzft3btXks75OevXr9eYMWMkSb6+vvr0008llf3zKDRlyhR9++23zp93dHS0atSooQ8//LDEsg8++KD69++vzZs3a/PmzXrjjTf0xhtv6KOPPpKvr6+kCzvlb8CAAZKsMdGuXTulpKTot99+U8+ePZ3r6Nu3r6ZPn17mdZ6S1KVLF+f66tatq4yMDKWkpKhbt27y8fGRZP2OePvttyVJERERmjp1qlavXq22bdvq8ccfd66rdu3aqlmzpn799VfdfPPN5doeAFUPgQpApbrxxhu1cuVKbdy4URs2bNCDDz6oqVOnqnPnzud8T40aNYpNl3Y60Zo1azR9+nQ9+OCD6tKlixo1aqRPPvmkxHIOh0Nt2rRxhhXJOu2r8FQwY4xz/rlOL7r++uvVrFkzrVmzRkuWLNGECRMkSZ06ddLy5cu1fv16paSkaN68eVqyZIlzB1WSdu7cqZEjR+qBBx5Qu3btdNddd2nKlCmlbteRI0ecpwkVcnNzK1ZjXl6e89+LFi3SBx98oJiYGEVFRal27drl2kE883ONMXJ3dz9vnZLk7e3t/Pff//53bd++Xffee69CQkKUn59frMYzf35eXl4lPt/hcMgYo/fee0+1atWSJB07dkw1a9Ysddm5c+c6dz5PnDghNze3EnW5u7vrnXfe0Y4dO5SSkqJnn31W7du319ixY4utz83NrViPu3btqrlz5yotLU3t2rVT3bp1tXbtWn311VcaNWpUGZ0s6mVhTWf2QbLG1Jn1SlbIKigoKDHfGKP8/Hx5eXkVu77r7DFwLmcuU1hXQUGBbr755mJBJjU1VX5+fkpMTDzn53h6ehar7+DBg6pTp06ZP49CkyZNklT2KX/ffPONtm7dqvj4eHXq1EmdOnXS448/rsjISK1bt049evQoc7vPduYfNRwOhzw9PUv8fyUV9ft8/49JKjYuz1z2XL87BgwYoE6dOmndunX6+uuv9corrygpKcm5Hg8PD1unMgKoOrjLH4BKtWjRIo0fP1533323xowZo7vvvlu7du2SZO1Y5Ofn21rvunXr1KlTJw0cOFBBQUFKTk5WQUFBifW2adNG69at03//+19J0pdffql77rlHOTk5at++vT766CM5HA5lZGRo1apV5/y8//f//p/eeOMNZWdnO6+z+tvf/qZly5apV69emjRpknx8fHTgwIFi79u8ebOCgoL04IMPqnXr1lq1apWzzjZt2ujTTz9Vbm6uHA6HJk+erM8++0weHh7Onbo//OEPysvLc94B8bPPPnOue+3aterTp4/69++vhg0bavXq1c51n88nn3zi3ObPP/9cnTt3Pm+dZ1u7dq3i4uLUu3dv1a1bV+vXry/X5xby8fFRy5Yt9a9//UuStVN+3333Oft/5s/v7rvv1r///W8ZY5Sbm6u//OUveuedd0qsc8+ePYqMjNTNN9+s4cOH64EHHij1BhINGzbUwYMHndN33HGHDh48qDVr1qht27Zq166d/vOf/6hBgwbFjjRJVsgoKCgoV7gp1KhRI7m5uTlvwLJz507FxcXp9ttv188//6zt27dLkn788Udt3rxZrVu3Lve6C9WpU0fNmzd3hqadO3fqhx9+kGQdWfzll1+0efNmSdZdAsPDw8u8aUubNm20ePFiSVJmZqbi4uK0f//+cv88ysvPz0/z58/Xli1bnPPS09OVlZWlW2+91dY6ly5dKsm6uUtKSoratGmj9u3ba9myZc67Mi5evNh57VOdOnWcd5VMTU3Vpk2byvyMsLAwJSUl6cSJE3I4HM4bl0hWoNq9e7f69u2rZ555RidOnFB6erokq5e5ubm67rrrbG0bgKqBI1QALom4uLgSp7s9/vjj6t27tzZt2qSePXuqVq1auvbaa523F+7Ro4diY2Nt3ap8wIAB+tvf/qaoqCjl5+erXbt2WrFihRwOh1q2bKl58+bpr3/9q1555RVNnTpVjz/+uIwxzhsjXH311Xr44Yc1adIkRUREyM/P77w7cJ07d9aUKVM0dOhQ57yRI0dq4sSJev/99+Xh4aGuXbvqrrvuKva+yMhIrVixQhEREXI4HOrUqZMyMjKUlZWlAQMG6Ndff1Xfvn1ljFHr1q0VGxurrKws1axZU/369dOHH36oMWPGaOjQofLz8yv2F/vBgwfr6aefdp5+1bJlS+eO9Pnk5OSoX79+OnnypAYOHKg2bdrolltuOWedZ3vooYc0c+ZMzZ07V15eXrrjjjtKBMmyzJo1S88884yioqKUm5uryMhI3XPPPZKs0/DGjBmjyZMna+LEiZo+fbqioqKUl5entm3bKj4+vsT6mjRpooiICN17773y9vbWVVddpSeffLLEcl27dtWbb77pvOGBu7u7wsLCtGPHDvn5+enOO+9URkZGqaf7+fv7q0WLFurVq5cWLlxYru2sUaOGXn75ZT377LOaOXOmvLy89PLLL6tu3bqaO3eunnnmGeXk5MjNzU0zZsxQw4YNnTcguRAvvviixo8fr/fee0833XST88Ykfn5+eumllzRz5kydPn1axhjNnDlTN9xww3mDw9NPP63JkycrKipKxhgNHz5cQUFB5f55FOrbt6/69u17ztcbNmyoefPmafbs2frtt99Us2ZN+fr66tlnny12c5Vz/X7p0KFDiXWePn1affr0UV5enp588kk1bNhQDRs21AMPPKC4uDg5HA75+fnptddek7u7u2JjYzV69GiFh4frhhtuUGho6DnrLdShQwft3btX9957r/7whz+oSZMmOn78uCRp9OjRevbZZzVnzhy5ubnpr3/9q2644QZJ1h8jOnbsWOIoPIDqxc1cyJ/WAACXldjYWMXExNg6lepy8dRTT6lNmzYlrtcCLrVBgwZpwoQJzusKAVRPnPIHALiijRkzRh9++GG5H+wLVISVK1cqODiYMAVcBjhCBQAAAAA2cYQKAAAAAGwiUAEAAACATQQqAAAAALCJQAUAAAAANlXL51AdP35SDkfVuZdG3bo+Onq05LNZrjT0wUIfLPShCL2w0AcLfbDQBwt9sNCHIvTCUpX64O7upjp1rj7n69UyUDkcpkoFKklVrh5XoQ8W+mChD0XohYU+WOiDhT5Y6IOFPhShF5bq0gdO+QMAAAAAmwhUAAAAAGATgQoAAAAAbCJQAQAAAIBNBCoAAAAAsIlABQAAAAA2EagAAAAAwCYCFQAAAADYRKACAAAAAJsIVAAAAABgE4EKAAAAAGwiUAEAAACATQQqAAAAALCJQAUAAAAANhGoAAAAAMAmAhUAAAAA2ESgAgAAAACbCFQAAAAAYBOBCgAAAABsIlBVgI3JCa4uAQAAAIALeLq6gOrms59XaNn+5GLzwrbl6IWjy4vN69mgq3o16l6ZpQEAAACoZASqC9SrUfcSQen9bY9oXueZLqoIAAAAgKsQqAAAAAAXKO3Mp9Jw5lPVRqACAAAAXKC0M5/8/X2Vnp7poopgBzeluAjZWRnaumaJJGnfzo0urgYAAABAZSNQ2VAYpJIWzJAkRQ8Zoz2bk7Ux6R3l5512cXUAAAAAKguByob9uzZJknrETVCrjn11Y+Nm6hYzWpK0cuEsZRw57MryAAAAAFQSrqGyoWnrbiXmeXrVVEiP+7Vv50Yd2rdTf7zmWhdUBgAAAKAyEagqWMPmIa4uAQAAANXUxuQENbq9i6vLwAXglD8AAACgiti86mNXl4ALRKACAAAAAJsIVAAAAABgE4EKAAAAAGwiUAEAAAAuVvicU0nat3Oji6vBhShXoEpMTFTPnj3VvXt3LVy4sMTrycnJio6O1j333KORI0cqIyNDknTo0CHFxMSoR48e+stf/qKTJ09Kkk6cOKFhw4YpIiJCMTExSk9Pr8BNAgAAAKqHwiCVtGCGJCl6yBjt2ZysjUnvKD/vtIurQ3mUGahSU1M1e/ZsLVq0SAkJCXr//ff1008/OV/PysrS5MmT9frrr+uTTz5RYGCgXn75ZUnSlClTNHDgQCUlJSkoKEivvvqqJGnOnDkKDg7W559/rv79+2v69OmXaPMAAACAqmv/rk2SpB5xE9SqY1/d2LiZusWMliStXDhLGUcOu7I8lEOZgWr9+vUKDQ1V7dq15e3trfDwcCUlJTlfz8vL06RJk1SvXj1JUmBgoA4fPqy8vDxt3rxZ4eHhkqS+ffs637dmzRpFRUVJkiIjI/XVV18pLy+vwjcOAAAAqMqatu6mVh37qtbVf3DO8/SqqZAe96vJXV11aN9OF1aH8ijzwb5paWny9/d3TgcEBGj79u3O6Tp16qhbt26SpJycHL3++uuKjY3V8ePH5ePjI09P6yP8/f2VmppaYp2enp7y8fHRsWPHnKGsLHXr+pRz8yqPv7+vq0uoEuiDhT5Y6EMRemGhDxb6YKEPFvpgoQ9FCnvh37GriytxreoyJsoMVA6HQ25ubs5pY0yx6UKZmZl66KGH1KRJE/Xp00epqakllivtfYXrdHcv//0xjh7NksNhyr38pebv76v09ExXl+Fy9MFCHyz0oQi9sNAHC32w0AcLfbDQhyL0wlKV+uDu7nbeAzplppj69esXu2lEenq6AgICii2TlpamgQMHKjAw0Hk9lJ+fnzIzM1VQUFDifQEBATpy5IgkKT8/XydPnlTt2rUvcNMAAAAAwLXKDFRt27ZVSkqKjh07puzsbK1YsUJhYWHO1wsKCjRixAhFRERo4sSJzqNQXl5eCg4O1rJlyyRJCQkJzvd16NBBCQkJkqRly5YpODhYXl5eFb5xAAAAAHAplXnKX7169TRq1CgNGjRIeXl56tevn1q0aKGhQ4fqkUce0W+//aZdu3apoKBAy5cvlyQFBQVp+vTpmjRpksaNG6f58+fr2muv1YsvvihJevTRRzVu3Dj16tVLvr6+mjVr1qXdSgAAAAC4BNyMMVXnYqRy4hqqqok+WOiDhT4UoRcW+mChDxb6YKEPFvpQhF5YqlIfLvoaKgAAAABA6QhUAAAAAGATgQoAAAAAbCJQAQAAAIBNBCoAAAAAsIlABQAAAAA2EagAAAAAwCYCFQAAAADYRKACAAAAAJsIVAAAAABgE4EKAAAAAGwiUAEAAACATQQqAAAAALCJQAUAAAAANhGoAAAAAMAmAhUAAAAA2ESgAgAAAACbCFQAAAAAYBOBCgAAAABsIlABAAAAgE0EKgAAAACwiUAFAAAAADYRqC7Q6W3LlH9o93mXyT+0W6e3LaukigAAAFAdsV95eSBQXSCPgIbKSX71nIM//9Bu5SS/Ko+AhpVcGQAAAKoT9isvDwSqC+R5XVNd1XVkqYO/cNBf1XWkPK9r6qIKAQAAUB2wX3l5IFDZUNrgZ9ADAADgQrFfWf15urqA6urMwX/s9/8qZ8tyBj0AAAAuGPuV1RtHqC6C53VN5dWsk35f+5G8mnVi0AMAAMAW9iurLwLVRcg/tFt5u75Q7bv7KW/XF2XepQUAAAAoDfuV1ReByqYzz23163DfOS8oBAAAAM6H/crqjUBlQ2kXCp7vLi0AAABAadivrP4IVBfofHddYfADAACgvNivvDwQqC5QQdq+8951pXDwF6Ttq+TKAAAAUJ2wX3l54LbpF6hmy55lLuN5XVPuzAIAAIDzYr/y8sARKgAAAACwiUBVARYt3+PqEgAAAAC4AIGqAry7Yq+rSwAAAADgAgQqAAAAALCJQAUAAAAANhGoAAAAAMAmbpt+gRK+/lmfrNtfYv7g51YXm76nXQP1bt+okqoCAAAA4AoEqgvUu32jEkFp8HOr9da4zi6qCAAAAICrcMofAAAAANhEoAIAAACqCJ5vWv0QqAAAAIAqguebVj8EqgpwX/dAV5cAAAAAwAUIVBVgYHgTV5cAAAAAwAUIVAAAAABgE7dNBwAAAFyA55teHghUAAAAgAvwfNPLA6f8AQAAAIBNBCoAAAAAsIlABQAAAAA2EagAAACAKoLnm1Y/BCoAAACgiuD5ptVPue7yl5iYqPnz5ys/P19xcXGKiYkpdbmxY8cqNDRUffv21dGjRzV48GDna5mZmTp+/Li2bt2qTZs26eGHH1b9+vUlSc2aNdOMGTMqYHMAAAAAoPKUGahSU1M1e/ZsLVmyRDVq1NCAAQMUEhKixo0bF1tm0qRJSklJUWhoqCSpbt26+vjjjyVJDodDcXFxGjVqlCTp+++/1+DBgzV8+PBLsU0AAAAAUCnKPOVv/fr1Cg0NVe3ateXt7a3w8HAlJSUVWyYxMVFdunRRREREqetYvHixatWqpaioKEnSjh07tHbtWkVFRWnEiBE6fPhwBWwKAAAAAFSuMgNVWlqa/P39ndMBAQFKTU0ttkx8fLz69+9f6vsLCgr0j3/8Q3/729+c83x9fRUbG6vExER16NDBeeQKAAAAAKqTMk/5czgccnNzc04bY4pNl+Xrr79WgwYNFBhYdMeSqVOnOv9933336YUXXlBmZqZ8fX3Ltc66dX3K/fmVxd+/fLVf7uiDhT5Y6EMRemGhDxb6YKEPFvpgoQ9F6IWluvShzEBVv359bdmyxTmdnp6ugICAcn9AcnKyevbs6Zx2OBx67bXXNGzYMHl4eDjnn/nvshw9miWHw5R7+UvN399X6emZri7D5eiDhT5Y6EMRemGhDxb6YKEPFvpgoQ9F6IWlKvXB3d3tvAd0yjzlr23btkpJSdGxY8eUnZ2tFStWKCwsrNwFbNu2TcHBwWcU5K6VK1dq+fLlkqSEhATdfvvt8vb2Lvc6AQAAAKAqKDNQ1atXT6NGjdKgQYPUu3dvRUZGqkWLFho6dKh27NhR5gccPHjQeXv0Qs8//7wWLFigXr16afHixZo2bZr9LQAAAAAAF3EzxlSdc+fKiVP+qib6YKEPFvpQhF5Y6IOFPljog4U+WOhDEXphqUp9uOhT/gAAAAAApSNQAQAAAIBNBCoAAAAAsIlABQAAAAA2EagAAAAAwCYCFQAAAADYRKACAAAAAJsIVAAAAABgE4EKAAAAAMvadMoAACAASURBVGzydHUBAAAAwJXoyMdLdSzx42LzfihlOb+oaF0T3adyisIFI1ABAAAALnBNdJ8SQcnf31fp6Zkuqgh2EKguEH9JAAAAAFCIQHWB+EsCAAAAgELclAIAAAAAbCJQAQAAAIBNBCoAAAAAsIlABQAAAAA2EagAAAAAwCYCFQAAAADYRKACAAAAAJsIVAAAAABgE4EKAAAAAGwiUAEAAACATQQqAAAAALCJQAUAAAAANhGoAAAAAMAmAhUAAAAA2ESgAgAAAACbCFQAAAAAYBOBCgAAAABsIlABAAAAgE0EKgAAAACwiUAFAAAAADZ5urqA6mjrhgM6lZWrlqE36mqfms75J7NOa9uGg/L2qaFWoTe5sEIAAAAAlYEjVDYEBtWX3KT339yidck/6fD/MrQu+Se9/+YWye3/XgcAAABw2eMIlQ3ePjXUrktjtQq5SVs3HtAbs79Wi7uu14D4u+TtU8PV5QEAAACoJByhugiFwUqS2nVpTJgCAAAArjAEKgAAAACwiUAFAAAAADYRqAAAAADAJm5KUQHCut/i6hIAAABQDfE4nuqPI1QVoGN4oKtLAAAAQDXE43iqP45QAQAAAC7C43iqP45QAQAAAC7G43iqLwIVAAAAANhEoAIAAAAAmwhUAAAAAGATgQoAAACoIngcT/VDoAIAAACqCB7HU/0QqAAAAADAJgIVAAAAANhEoAIAAAAAmwhUAAAAAGATgQoAAAAAbCJQAQAAAIBNBCoAAAAAsKlcgSoxMVE9e/ZU9+7dtXDhwnMuN3bsWC1ZssQ5vXTpUt19992Kjo5WdHS0Zs+eLUk6ceKEhg0bpoiICMXExCg9Pf0iNwMAAAAAKp9nWQukpqZq9uzZWrJkiWrUqKEBAwYoJCREjRs3LrbMpEmTlJKSotDQUOf877//XuPGjVNkZGSxdc6ZM0fBwcF6/fXXlZCQoOnTp2vOnDkVuFkAAAAAcOmVeYRq/fr1Cg0NVe3ateXt7a3w8HAlJSUVWyYxMVFdunRRREREsfk7duzQ0qVLFRUVpdGjRysjI0OStGbNGkVFRUmSIiMj9dVXXykvL6+itgkAAAAAKkWZR6jS0tLk7+/vnA4ICND27duLLRMfHy9J+uabb4rN9/f31+DBg3XHHXfoxRdf1NSpU/XCCy8UW6enp6d8fHx07Ngx1atXr1xF163rU67lKpO/v6+rS6gS6IOFPljoQxF6YaEPFvpgoQ8W+mChD0XohaW69KHMQOVwOOTm5uacNsYUmz6fefPmOf8dHx+vbt26lbqcMUbu7uW/P8bRo1lyOEy5l7/U/P19lZ6e6eoyXI4+WOiDhT4UoRcW+mChDxb6YKEPFvpQhF5YqlIf3N3dzntAp8wUU79+/WI3jUhPT1dAQECZH5yZmal///vfzmljjDw8PCRZR7mOHDkiScrPz9fJkydVu3btMtcJAAAAAFVJmYGqbdu2SklJ0bFjx5Sdna0VK1YoLCyszBV7e3vrzTff1HfffSdJeuedd5xHqDp06KCEhARJ0rJlyxQcHCwvL6+L2Q4AAAAAqHRlnvJXr149jRo1SoMGDVJeXp769eunFi1aaOjQoXrkkUd02223lfo+Dw8PzZkzR5MnT1ZOTo4aNGigmTNnSpIeffRRjRs3Tr169ZKvr69mzZpVsVsFAAAAAJXAzRhTdS5GKieuoaqa6IOFPljoQxF6YaEPFvpgoQ8W+mChD0XohaUq9eGir6ECAAAAAJSOQAUAAAAANhGoAAAAAMAmAhUAAAAA2ESgAgAAAACbCFQAAAAAYBOBCgAAAABsIlABAAAAgE0EKgAAAACwiUAFAAAAADYRqAAAAADAJgIVAAAAANhEoAIAAAAAmwhUAAAAAGATgQoAAAAAbCJQAQAAAIBNBCoAAAAAsIlABQAAAAA2EagAAAAAwCYCFQAAAADYRKACAAAAAJsIVAAAAABgE4EKAAAAAGwiUAEAAACATQQqAAAAALCJQAUAAAAANhGoAAAAAMAmAhUAAAAA2ESgAgAAAACbCFQAAAAAYBOBCgAAAABsIlABAAAAgE0EKgAAAACwiUAFAAAAADYRqAAAAADAJgIVAAAAANhEoAIAAAAAmwhUAAAAAGATgQoAAAAAbCJQAQAAAIBNBCoAAAAAsIlABQAAAAA2EagAAAAAwCYCFQAAAADYRKACAAAAAJsIVAAAAABgE4EKAAAAAGwiUAEAAACATQQqAAAAALCJQAUAAAAANhGoAAAAAMAmAhUAAAAA2ESgAgAAAACbCFQAAAAAYBOBCgAAAABsKlegSkxMVM+ePdW9e3ctXLjwnMuNHTtWS5YscU5/88036tevn6KjoxUXF6dff/1VkrRp0yaFhIQoOjpa0dHRGj9+/EVuBgAAAABUPs+yFkhNTdXs2bO1ZMkS1ahRQwMGDFBISIgaN25cbJlJkyYpJSVFoaGhzvljxozRq6++qiZNmuijjz7StGnTNH/+fH3//fcaPHiwhg8ffmm2CgAAAAAqQZlHqNavX6/Q0FDVrl1b3t7eCg8PV1JSUrFlEhMT1aVLF0VERDjn5ebm6tFHH1WTJk0kSYGBgTp8+LAkaceOHVq7dq2ioqI0YsQI53wAAAAAqE7KPEKVlpYmf39/53RAQIC2b99ebJn4+HhJ1il+hWrUqKHo6GhJksPh0CuvvKKuXbtKknx9fRUREaHu3bvr3Xff1ahRo/Tee++Vu+i6dX3KvWxl8ff3dXUJVQJ9sNAHC30oQi8s9MFCHyz0wUIfLPShCL2wVJc+lBmoHA6H3NzcnNPGmGLTZcnNzdW4ceOUn5/vPMVv6tSpztfvu+8+vfDCC8rMzJSvb/madvRolhwOU+4aLjV/f1+lp2e6ugyXow8W+mChD0XohYU+WOiDhT5Y6IOFPhShF5aq1Ad3d7fzHtAp85S/+vXrKz093Tmdnp6ugICAcn34yZMnFR8fr/z8fM2fP19eXl5yOByaP3++CgoKii3r4eFRrnUCAAAAQFVRZqBq27atUlJSdOzYMWVnZ2vFihUKCwsr18rHjBmjP/3pT5ozZ45q1KhhfaC7u1auXKnly5dLkhISEnT77bfL29v7IjYDAAAAACpfmaf81atXT6NGjdKgQYOUl5enfv36qUWLFho6dKgeeeQR3XbbbaW+b9euXVq1apUaN26sPn36SLKuv3rjjTf0/PPP66mnntK8efPk5+enmTNnVuxWAQAAAEAlcDPGVJ2LkcqJa6iqJvpgoQ8W+lCEXljog4U+WOiDhT5Y6EMRemGpSn246GuoAAAAAAClI1ABAAAAgE0EKgAAAACwiUAFAAAAADYRqAAAAADAJgIVAAAAANhEoAIAAAAAmwhUAAAAAGATgQoAAAAAbCJQAQAAAIBNBCoAAAAAsIlABQAAAAA2EagAAAAAwCYCFQAAAADYRKACAAAAAJsIVAAAAABgE4EKAAAAAGwiUAEAAACATQQqAAAAALCJQAUAAAAANhGoAAAAAFQpG5MTXF1CuRGoAAAAAFQpm1d97OoSyo1AhQpTnf6SAAAAAFQEAhUqTHX6SwIAAABQEQhUAAAAAGATgQoAAABAlZCdlaGta5ZIkvbt3OjiasqHQAUAAADApQqDVNKCGZKk6CFjtGdzsjYmvaP8vNMuru78CFS4aNXxLwkAAACoOvbv2iRJ6hE3Qa069tWNjZupW8xoSdLKhbOUceSwK8s7L09XF4DqKzsrQ3u2rNL+XZvUoFlrRQ8Zoy+WLlDawR91Z5f+8vSq6eoSAQAAUA00bd2txDxPr5oK6XG/9u3cqEP7duqP11zrgsrKxhEq2Fad/5IAAACA6qFh8xA1vaurq8s4J45Qwbbq/JcEAAAAoCIQqHBJNGwe4uoSAABANbExOUGNbu/i6jIAWzjlDwAAAC61edXHri4BsI1ABQAAAAA2EagAAAAAwCYCFQAAAADYRKACAACAS2RnZWjrmiWSpH07N7q4GsAeAhUAAAAqVWGQSlowQ5IUPWSM9mxO1sakd5Sfd9rF1QEXhkAFAACASrV/1yZJUo+4CWrVsa9ubNxM3WJGS5JWLpyljCOHXVkecEF4DhVwET77eYWW7U8uc7meDbqqV6PulVARAABV22c/r9CyrP/77ty4ofiLNaR6V+dr28q/q0VID747US0QqGALQcLSq1H3Etvn7++r9PRMF1UEAEDVxncnzlbd9ysJVLCFX4bnxtPeAQAAyq+671dyDRVQwXjaOwAAwJWDQAUAAAAANhGoAAAAAMAmAhUAAAAA2ESgAioIT3sHAAC48hCogIvE094BAACuXAQq4CLxtHcAAIArF8+hAi5S09bdSszz9KqpkB73a9/OjTq0b6f+eM21LqgMAAAAlxqBCriEGjYPcXUJAAAAuIQ45Q8AAAAAbCJQAQAAAIBNBCoAAAAAsIlABQAAAAA2lStQJSYmqmfPnurevbsWLlx4zuXGjh2rJUuWOKcPHTqkmJgY9ejRQ3/5y1908uRJSdKJEyc0bNgwRUREKCYmRunp6Re5GQAAAABQ+coMVKmpqZo9e7YWLVqkhIQEvf/++/rpp59KLDNixAgtX7682PwpU6Zo4MCBSkpKUlBQkF599VVJ0pw5cxQcHKzPP/9c/fv31/Tp0ytwkwAAAACgcpQZqNavX6/Q0FDVrl1b3t7eCg8PV1JSUrFlEhMT1aVLF0VERDjn5eXlafPmzQoPD5ck9e3b1/m+NWvWKCoqSpIUGRmpr776Snl5eRW2UQAAAABQGcoMVGlpafL393dOBwQEKDU1tdgy8fHx6t+/f7F5x48fl4+Pjzw9rUdd+fv7O9935jo9PT3l4+OjY8eOXdyWAAAAAEAlK/PBvg6HQ25ubs5pY0yx6XMpbblzvc8YI3f38t8fo25dn3IvW1n8/X1dXUKVQB8s9MFCH4rQCwt9sNAHC32w0AcLfShCLyzVpQ9lBqr69etry5Ytzun09HQFBASUuWI/Pz9lZmaqoKBAHh4exd4XEBCgI0eOqH79+srPz9fJkydVu3btchd99GiWHA5T7uUvNX9/X6WnZ7q6DJejDxb6YKEPReiFhT5Y6IOFPljog4U+FKEXlqrUB3d3t/Me0CnzsFDbtm2VkpKiY8eOKTs7WytWrFBYWFiZH+zl5aXg4GAtW7ZMkpSQkOB8X4cOHZSQkCBJWrZsmYKDg+Xl5VWuDQIAAACAqqLMQFWvXj2NGjVKgwYNUu/evRUZGakWLVpo6NCh2rFjx3nfO2nSJH3wwQfq2bOntmzZoscee0yS9Oijj2rbtm3q1auXFi1apKeffrpitgYAAAAAKpGbMabqnDtXTpzyVzXRBwt9sNCHIvTCQh8s9MFCHyz0wUIfitALy8pvf1W3O653dRmSKuCUP6C8Fi3f4+oSAAAAcBl4d8VeV5dQbgQqVJjqNPABAACAikCgAgAAAACbCFQAAAAAYFOZz6ECAAAAgEsl4euf9cm6/SXmD35udbHpe9o1UO/2jSqpqvIjUMGW6j7wAQAAUDX0bt+oxP7i4OdW661xnV1U0YUhUMGWCN89ihzUUJ7XNXXOO3vg5x/arYK0PZIu30B1etsyeQQU78PZrD7sU82WPSuxMgAAqia+O3G26j4muIYKtngENFRO8qvKP7S71NfzD+1WTvKr8ghoWMmVVS76AADAheG7E2er7mOCQAVbPK9rqqu6jix18BcO+qu6jjzvXxouB/QBAIALw3cnzlbdxwSBCraVNvirw6CvaPQBAIALw3cnznb2mLive2C1GRNcQ4WLcubgH9M8WDnJS6v8oL8UzuzDsd//q5wty6/IPgAAUF5nfnd+9c0m3Xh8C9+dV7gzx0SP4HBlJFeP/SmOUOGieV7XVF7NOumGw1/Iq1mnKj/oL5XCPvy+9qMrug8AAJQX+xA4W3XcnyJQ4aLlH9qtvF1fqPbd/ZS364tzXlB4uaMPAABcmMLvzqTsFnx3QlL13J8iUOGinHluq1+H+855QeHljj4AAHBhzvzu/Dy7Jd+dqLb7UwQq2FbahYLnu0vL5Yo+AABwYfjuxNmq85ggUMGW8911pboM/opAHwAAuDBfJq1S+idz9FJaGw1bcFiDn1stSRr83GoNW3BYL6W1Ufonc/Rl0ioXV4rKUt33p7jLH2wpSNt33ruuFA7+grR91eJiQrvoAwAAFya0/ml5tHhME874Xhz83Gq9Na6zczr/UCv5pe1zRXlwgeq+P0Wggi01W/YscxnP65pWyUFfkegDAAAXhu9OnK26jwlO+QMAAAAAmwhUQAVbtHyPq0sAAABAJSFQARXs3RV7XV0CAADVyn3dA11dAmAbgQoAAAAuNTC8iatLAGwjUAEAAACATQQqAAAAALCJ26YDFyHh65/1ybr9JeYXPqSw0D3tGqh3+0aVVBUAAAAqC4EKuAi92zcqEZTOfjghAAAALl+c8gcAAAAANhGoAAAAAMAmAhUAAAAA2ESgAioYDycEAAC4chCogArGwwkBAACuHAQqAAAAALCJQAUAAAAANhGoAAAAAMAmAhUAAAAA2ESgAgAAAACbCFQAAAAAYBOBCgAAAABsIlABAAAAgE0EKgAAAACwiUAFAAAAADYRqAAAAADAJgIVAAAAANhEoAIAAAAAmzxdXQAAAACAK9vWDQd0KitXLUNv1NU+NZ3zT2ad1rYNB+XtU0OtQm9yYYXnRqCCLUc+Xqpvv/5ZuZ619KfjO1SzIFs//N9rpz1q6Zc6t6lGfrbuaN9I10T3cWmtl9KRj5fqWOLHxeb9UMpyflHRl3UfAAAoL747UZrAoPrauvGA3n9ziwKD6imkfSNt/Ppn7f0+VYG31VNgUH1Xl3hObsYY4+oiLtTRo1lyOKpO2f7+vkpPz3R1GZXuVFautm48oL07Uksd+K1CbpK3Tw1Xl1nprtTxcDb6UIReWOiDhT5Y6IOFPljoQ5ErsRdnhmzrD/NBOlg7SDf+/r3+dPx71SzIluS6kO3u7qa6dX3O+TpHqGCbt08NtevSWK1CbtLWjQf0xuyv1eKu6zUg/q4rMkgBAADgwl0T3adYULpN0vznvlTkcw+5rqgLwE0pcNEKg5UktevSmDAFnGVjcoKrSwAAAJcIgQoALrHNqz4ueyEAAFAtEagAAKhEHLEEgMsLgQoAgErEEUsAKFtY91tcXUK5EahQYarTwAcAAEDV1TE80NUllBuBChWmOg18oDJkZ2Vo65olkqR9Oze6uBoAAHApEKgAoIIVBqmkBTMkSdFDxmjP5mRtTHpH+XmnXVwdAACoSAQqAKhg+3dtkiT1iJugVh376sbGzdQtZrQkaeXCWco4ctiV5cFFOGIJAJencj3YNzExUfPnz1d+fr7i4uIUExNT7PXdu3dr4sSJOnnypIKDgzVlyhRlZGRo8ODBzmUyMzN1/Phxbd26VZs2bdLDDz+s+vXrS5KaNWumGTNmVOBmAYDrNG3drcQ8T6+aCulxv/bt3KhD+3bqj9dc64LK4ArZWRnas2WV9u/apAbNWit6yBh9sXSB0g7+qDu79JenV01XlwhUuq0bDuhUVq5aht6oq32K/h84mXVa2zYclLdPDbUKvcmFFQLlV2agSk1N1ezZs7VkyRLVqFFDAwYMUEhIiBo3buxcZsyYMZo2bZpatmypCRMm6IMPPtDAgQP18cfWnYwcDofi4uI0atQoSdL333+vwYMHa/jw4ZdoswCgamrYPMTVJaCSnXnEstbVf5C/v6+6xYzWN6s+1MqFs9Q2cjABG1ecwKD62rrxgN5/c4sCg+oppH0jbfz6Z+39PlWBt9VTYFB9V5cIlFuZp/ytX79eoaGhql27try9vRUeHq6kpCTn67/++qtycnLUsmVLSVLfvn2LvS5JixcvVq1atRQVFSVJ2rFjh9auXauoqCiNGDFChw9z+gsA4PLUtHU3terYV7Wu/oNzXuERyyZ3ddWhfTtdWB3gGt4+NdSuS2MNiL9LcpPemP215CYNiL9L7bo0lrdPDVeXCJRbmUeo0tLS5O/v75wOCAjQ9u3bz/m6v7+/UlNTndMFBQX6xz/+oVdffdU5z9fXVxEREerevbveffddjRo1Su+99165i65b16fcy1YWf39fV5dQJdAHC32w0Ici9MJCHyyFffDv2NXFlbgW48FyRffBX/pTw7ravvlX9R7QytXVVBlX9Jg4Q3XpQ5mByuFwyM3NzTltjCk2XdbrX3/9tRo0aKDAwKJbak+dOtX57/vuu08vvPCCMjMz5etbvqYdPZolh8OUa9nK4O/vq/T0TFeX4XL0wUIfLPShCL2w0AcLfbDQBwt9KEIfLIwJS1Xqg7u723kP6JR5yl/9+vWVnp7unE5PT1dAQMA5Xz9y5Eix15OTk9WzZ0/ntMPh0Pz581VQUFDsczw8PMoqBQAAAACqlDIDVdu2bZWSkqJjx44pOztbK1asUFhYmPP166+/XjVr1tQ333wjSfr444+Lvb5t2zYFBwcXfaC7u1auXKnly5dLkhISEnT77bfL29u7wjYKAAAAACpDmYGqXr16GjVqlAYNGqTevXsrMjJSLVq00NChQ7Vjxw5J0qxZszRjxgz16NFDp06d0qBBg5zvP3jwoPP26IWef/55LViwQL169dLixYs1bdq0Ct4sAAAAVBdh3W9xdQmAbW7GmKpzMVI5cQ1V1XSl9uHsZ2kU9uFKf5bGlToeSkMvLPTBQh8s9MFCHyz0oQi9sFSlPpR1DVW5HuwL4Nx4lgaAc/ns5xVatj+5zOV6NuiqXo26V0JFAICKRqACLlLhszRahdykrRsP6I3ZX6vFXddrQPxdPEfjCsQONM7Uq1H3Ej/nqvRXVwDAxSNQARWkMFht3/yr2nVp7Opy4CLsQAMAcGUp86YUAAAAAIDSEagAAAAAwCYCFQAAAADYRKACKhjP0gAAALhyEKiACtYxPNDVJQAAAKCSEKgAAAAAwCYCFQAAAADYRKACAAAAAJsIVAAAAABgE4EKAAAAAGwiUAHAJbZo+R5XlwAAAC4RAhUAXGLvrtjr6hIAAMAlQqACAAAAAJsIVAAAVCJOAQWAywuBCgCASsQpoABwefF0dQEAcDlJ+PpnfbJuf4n5g59bXWz6nnYN1Lt9o0qqCgAAXCoEKgCoQL3bNyoRlAY/t1pvjevsoooAAMClxCl/AAAAAGATR6gAoAKd3rZMHgEN5Xld03Muk39otwrS9qlmy56VWBlc4duEt5X0k4d+yq9fbP6Zp4A29vxNPRoX6I7esZVdHgCgAhCoAKACeQQ0VE7yq7qq68hSQ1X+od3O13H5a9E6WLeeKD4ezjwF1BoPS3VVa8YDAFRXnPIHABXI87qmuqrrSOUkv6r8Q7slSfd1D5RUPEyd7wgWLh+ljYdCjAcAuDwQqACggp29Ez0wvAk7z1ew0kIV4wEALh+c8gcAl8CZO9HHfv+vcrYsZ+f5CnbmeIio1dA6zY/xAACXBY5QAcAl4nldU3k166Tf134kr2ad2Hm+whWOhx61tjMeAOAyQqACgEsk/9Bu5e36QrXv7qe8XV+UuIYGVxbGAwBcnghUAHAJnHmNjF+H+855YwJcGRgPAHD5IlABQAUr7YYD57vb2/9v7z4DoyjXPoxfSZZEkC6B0BGjocYQWg4IiDRpoQhIDwYUDgKKiodDOSBF4BxAsIAgIqKgIi2ANEGadBCkBFQ6CiEJJY20ze77YV8WYnpIsmH3//u0O/PMzD33zrO7987Ms2LfdDyIiNg3FVQiIjkovdHb9CXa8eh4EBGxfyqoRERyUFLoxXRHb7v3JTop9GIeRya2oONBRMT+adh0EZEc5ObTLsM2hnLVNcKbg9DxICJi/3SGSkREREREJJtUUImIiIiIiGSTCioREREREZFsUkElIiIiIiKSTSqoREREREREskkFlYiIiIiISDapoBIREREREckmFVQiIiIiIiLZpIJKREREREQkm1RQiYiIiIiIZJMKKhERERERkWxSQSUiIiIiIpJNKqhERERERESyyWDrAHKK2Wzm9u0wEhLiAHOebjs01BmTyZSn28yP8ioPLi4GChcuTsGCj+f6tkSy49iBK9yNTsDHryKPF3azTo+Jjuf4gasUKuxKHb9KNoxQREREcordFFTR0RE4OTlRpkwFnJzy9sSbweCM0aiCKi/yYDabSUxM4M6dMAAVVZIvedXy4NjBK3y36AhetcrQsElVDu65wG+nbuBVuwxetTxsHaKIiIjkELu55C82NpoiRYrneTElecvJyQlXVzeKF3cnOvqOrcMRSVWhwq40buFJz0H1wQk++2APOEHPQfVp3MKTQoVdbR2iiIiI5BC7qT5MpiRcXOzmhJtkoEABV5KSjLYOQyRd9worQIWUiIiInbKbggosZy+yY+2eCzkcieS27L7WIiJ57diBK+zddo6Y6Phk02Oi49m77RzHDlyxUWQiIpITdEoHWLf3Ep2bVM3RdV6/fo1evbpSpUry9Xbs2JmXXuqRov3evXu4evUyPXv2Ze3alQB07tztoWJ4cJ2ZNWzYawQGvoavb72H2raIowoPWsOt9UHJJ3q+wu+DBiSbVLJjJ0p16pJ3gYlNhAetwbBxK7dL1GL5QU/KRp2nbOQ5rhf15HqRpygbdY6it08RfqO1jgcRkUeUCqpcVKqUO0uWLM9U27Nng62PH7aQSm2dIpI3SnXqkuKL8fbpu3hm0RLbBCQ2de94qA3cjU7g2MGnOHS4Jt71y/NCw0oUKtzG1iGKiMhDUkGVx4xGI9OmvceFC+cB6NKlO7VrP0tQ0GoAPDzKEhJyHYCBAwfj79+GJk2aERx8ipIlS9G+vT8rV35LWFgoY8ZMoE6duhw7dpSFC+cRHx9HVFQ0I0aMpEKFSsnW2bx5S2bPnsGFC+cxmUz06dOfVq1eJCEhgRkzJnP27Bk8PMoREaGBHkRyWtPWT9s6BMkH7t1Td+Lw3EpZeQAAIABJREFUX9Z760RE5NGngioXhYeHMWBA72TT3njjbSIjI/nii+WEh4cxf/5H+Pt3oVOnrgC0b+/P558vsLa/desmfn6NGDVqDMOHD2b37h3Mm7eITZs2sGLFN9SpU5dVq75j9OjxVK5chaNHDzN37kyWLv0u2Trnz/8IL6/qjBv3HjEx0QwZEkiNGrXYtWsHAMuWreTq1SsEBPTKo+yIOI7n23gRFhZl6zBEREQkF2SqoFq/fj3z58/HaDQSEBBAnz59ks0/c+YMY8eOJSYmhnr16vHee+9hMBhYs2YNs2bN4oknngDg+eefZ+TIkURGRvLOO+9w9epVSpYsyZw5c3B3d8/5vUvF2j0XWLf3UorpgdN/Svbcv3GVh76vKrVL/iIi7nDlymXeemsYfn6Nef31NzJcj59fY8Bypsnb2weAMmU8iIqKBGD8+Mns27eHHTu2cfr0SWJjY1Os48iRQ8THx/HDD+sAiIuL4+LFCxw/fhR/f0vhVbFiJWrX9s7+DouIiIiIOJgMC6obN27wwQcfsHr1alxdXenZsycNGzbE0/P+5QqjRo1iypQp+Pj4MGbMGFasWEHv3r05deoUo0ePpkOHDsnWOWfOHOrVq8fChQtZu3YtU6dOZc6cOTm/d6no3KRqikIpcPpPLB79Qp5sv1ix4nz11QoOHz7I/v17CQzsy1dfrUh3mQIFClgfu7i4pJj/+uuv4utblzp16lK3bn3ee29cijYmUxLjx0/Gy6saYDnzVbRoMdatWwOY012/iIiIiIikLsNh0/ft24efnx/FixenUKFCtGnThs2bN1vn//XXX8TFxeHjYzlz0rVrV+v8kydPsmbNGjp27Mg777xDREQEADt37qRjx44AdOjQgd27d5OYmJjjO5cf/fzzLiZP/g+NGj3Hm2++Q8GCBQkNvYGLiwtJSUlZXl9kZARXr15m4MAh+Pk1Zs+eXZhMJoBk6/T1rW8dPTA8PJyAgF7cuBFCvXoN2Lp1MyaTiZCQ65w8eSLndlZERFLQPXUiIvYlwzNUoaGhyS7HK126NCdOnEhzvru7Ozdu3LA+DgwMxNfXl9mzZzNp0iRmzZqVbBmDwUDhwoW5desWZcqUybEdyw9Su4eqVq3auLm50a9fD1xdXWnTph1PPeVJVFQkU6dOpGTJklnaRtGixejQoRP9+vXAYDDg61ufuLg4YmNj8fHxta4zMPBVZs2aQb9+PTCZTAwdOoLy5SvQtWt3Ll48T58+3fDwKEvVqk/lZApERORvdE+diIh9cTKbzeb0GsyfP5/4+HjefPNNAFasWMGpU6eYNGkSAEePHmXWrFksX265V+jSpUsMGTIk2VksgIiICFq1asWhQ4eoVasWx48fx2Cw1HNNmjRh9erVD3Uf1enTwZQrVzlby/afso2l41pme9tiG9euXaZmzRq2DkNEREREHFiGZ6g8PDw4cuSI9XlYWBilS5dONj8sLMz6PDw8nNKlSxMVFcWqVasYMGAAAGaz2Xp/TunSpQkPD8fDwwOj0UhMTAzFixfPdNA3b0ZjMiWvA00mE0ajKdPreJB/4yrZXhbAYHB+qOXtRV7nwWQy5ctfed3di+TLuPKa8nCfcmGhPFgoDxbKg4XyYKE83KdcWOSnPDg7O/HEE4XTnp/RCho1asT+/fu5desWsbGxbN26laZNm1rnly9fHjc3N44ePQpAUFAQTZs2pVChQixatIhff/0VgK+//ppWrVoB0KxZM9auXQvAxo0bqVevXrKBF/Law47mJyIiIiIijinDM1RlypRh5MiR9O/fn8TERLp164a3tzevvvoqI0aMoHbt2sycOZNx48YRHR1NzZo16d+/Py4uLsyZM4eJEycSFxdHlSpV+O9//wvAG2+8wejRo2nfvj1FihRh5syZub6jIiIiIiIiOS3De6jyo9Qu+QsJuYyHR/buoXpYuuTPIq/zYMvXPD356RS1LSkP9ykXFsqDhfJgoTxYKA8WysN9yoVFfsrDQ1/yJyIiIiIiIqlzuIIq/vhGjNfOpNvGeO0M8cc35lFEIiIiIiLyqHK4gsql9JPEbZuXZlFlvHaGuG3zcCn9ZB5HJiIiIiIijxqHK6gM5arzWMuhqRZV94qpx1oOxVCueo5s78KFczz3XD127tyebrtr1/5i2rRJ2d7Oc8/Vy/ayIiIiIiKSPQ5XUEHqRVVuFFMAP/ywjubNWxIUtDrddiEh1/nrrz9zbLsiIiIiIpL7HLKgguRFVfyR1blSTBmNRrZu3cyrr/6T338/ay2YDh8+SEBAL/r3f5l3332TmJho5s6dyW+/nWHWrBn88ssRhg17zbqeqVMnsnHjegAWLPiE114bQM+eXRk27DVu3bqZY/GKiIiIiEjWOGxBBZaiqkCN5iT8so4CNZrnaDEFsG/fz3h4eFCpUmWaNHmeoKDVJCQkMGnSeMaNm8jSpd9RtaonmzZt4I033sHLqzpvv/2vNNf3559XuXLlEp9+uphvv11NmTIebNmyKUdjFhERERGRzHPogsp47QyJwTtw9fUnMXhHhqP/ZdXGjeto2bINAC1atGLjxvWcO/c77u7uPP20FwBDhgyjW7eemVpfhQoVGTZsJOvXr+Wjjz7g9OmTxMbezdGYRUREREQk8wy2DsBW/n7PlEu56jl62d/t27c4cGAfv/12lu+//xaz2UxUVCQHDuwDnKztoqOjuXs3JtmyTk5OyZ4bjUYAzp49w8SJY+nZszfNm7fAxcWZR/B/mUVERERE7IZDnqFKbQCK9Eb/y47NmzdSt24D1qzZyMqV61m1agP9+wdy4MA+7ty5zcWLFwBYtuxL1q5dhYuLgaSkJACKFSvOtWt/ER8fT2RkBL/+egyA48ePUqdOXTp37kbFipXYt+9nTCbTQ8cqIpIXDm5ba+sQREREcpzDFVTpjeaXk0XVpk3r6dKlW7JpXbv24Pz5Pxg/fhJTpkwgIKAnly5dpG/fAVSpUoXo6CgmTx5P1apP8Y9/NKZfvx6MHz+aZ5+tA0CLFq05d+53+vd/meHDB+PlVZ3r1689VJwiInnl8PYgW4cgIiKS45zMj+A1YzdvRmMyJQ87JOQyHh6VM1w2/vhGXEo/me5lfcZrZ0gKvYibT7tMxWMwOGM06kxRXuchs695XnN3L0JYWJStw7A55eE+5cLiu9kjePmtD20dhs3peLBQHiyUBwvl4T7lwiI/5cHZ2Yknniic5nyHu4cqM0WSoVz1HB/xT0RERERE7I/DXfInInlD98uISHr0HiEi9kIFlYjkCt0vI/fERkdwbOdqAC6ePmjjaCS/0HuEiNgLFVQiIpIr7hVSm5dOA6DTwFGcPbyNg5u/xpgYb+PoREREcoYKKhERyRWXgg8B8GLAGOo835WKnjVo1ecdAH5cNpOI8Ou2DE9ERCRHONygFCIikjeqN2iVYpqhgBsNX+zLxdMHuXbxNMVKlbVBZCIiIjlHZ6hyyVdfLaFXr64EBPTkyy8/t05///336NmzCwMG9GbAgN7s2rUDgE8+mUv37p2YNGm8te327VtZs2ZlnseeFcHBp5g3zzIM8u7du1i06FMAunXrqP/IclC6X0Yy48maDalev6WtwxAb0HuEiNgbFVS54PDhg2zbtplFi5ayePEygoNPsWvXTwCcPRvMxx9/xpIly1myZDnNmjUnKiqKAwf28v33QURFRXLu3B8YjUY2bdpAx46dbbw36bt06SK3b98CoGnTZgwaNMTGEYmt6H4ZEUmP3iNSp9EORR59uuQvF/zxx280aPAPHn/c8gdgDRs2YvfunTRs2IgbN0KYNm0S4eGhNG3anFdeeRUXFxdMJhPx8XHExcVRoEABVq/+ng4dOmEwpP4S/fLLEb788nNcXAxcv/4XNWrU5F//Go+rqyubNm3g+++/wWQy4+VVjbfe+hdubm506NASL68a3LwZzqJFS1m4cB67d+/EYHDB378rPXr04s8/rzJz5jQiIyNwc3uMkSNH8cwz1Zg6dSKPP16Y3347Q3h4GAMGDKJp0+YsWvQpsbGxfPnl55QpU4ajR48wduxEa5xJSUnMmzeXY8eOkpRkol27Drz8cp+8eBkkjz14v0zBx4vi7l6EVn3e4ej27/lx2UwadQjU5V0iDkzvEak7vD2Iqs+2sHUYIvIQdIYqFzzzTDUOHdpPZGQE8fHx/Pzzbm7dusnNm+H4+tbn3//+DwsWLOHXX4+xYUMQhQoVon17fwYO7Ef16jUpVaoUR44c4vnn03+DPXnyBG+++TbLl68iPj6B1atXcOHCedavX8v8+YtZsmQ5JUqU5JtvvgLgzp079OnTnyVLlrN7905OnvyVpUu/ZeHCL9m4cT03b4YzdeoEhg4dweLFy3j33bFMmDDGur3Q0BvMm7eI6dNn88kncylSpAiDBg3hueeaEhAwMNUY169fA8Dixcv47LMv2bNnF7/+eiyHMi35SfUGrajzfFcKPl7UOu3e/TLV6rfk2sXTNoxORGxN7xEiYq8c7gzVDxe2svHStgzbtavSkvZVW2drG/XqNaBt2w4MHz6YIkWKUq9eA4KDT1G+fAWmTZtpbffSSy+zefMP+Pt3oXfv/vTu3R+ABQs+oXfvfuzcuZ3164OoWLEiI0a8jbNz8vrXx6cOlSpVAeDFF9uxbt0aChQowJ9/XmXw4FcAMBoTeeaZatZlatasBcDx40d54YVWuLq64urqypIly7l79y5nzgTz/vuTrO1jY2OJiLgDQIMGDXFycqJq1aeIjIzIVC6OHDnEH3/8ztGjR/5/fXc5f/4czz5bJysplUfckzUb2joEEcnH9B4hIo8yhyuo2ldtne1CKbPu3o3h+edb0LNnXwCWL19KuXIVOH/+HFevXn7gzJM5xSV9YWGhXLv2Jz4+vvTs2ZUvv/yGuXNncuTIIRo08EvW1sXFxfrYZDLj4uJCUpKJF15oyZtvjvr/WO6SlJRkbefm9hgABoMBJ6f767p+/RpFihTF1dWNJUuWW6eHht6gaNFiALi6ugHg9OCCGUhKMjF06AiaNXsBsJwlK1iwYKaXFxERERHJz3TJXy64du0ao0e/jdFoJDo6mg0bgnjhhZaYzWY+/HA2kZGRGI1GgoLW0LTp88mWXbx4IQEBgwBISjLi4uKCs7MzCQkpb9g9ceI4YWGhmEwmNm/+gYYNG1GnTl12797J7du3MJvNzJo1jRUrlqdY9tlnfdm58yeMRiNxcXG8/fZwbt26SYUKFdmyZSMAhw8f4PXXX0t3Xy1FXFKa8+vWrce6dWsxGo3cvXuXoUMHcvr0yYxSKCIiYtc02qGI/XC4M1R5wdPzaZ5//gUGDOhFUlISL7/cG29vHwD69h3A0KEDMRqNNGv2Aq1avWhd7sKFc4DlkjqAbt1eplevl6hYsRINGvwjxXZKlXJnypQJhIWFUr9+Qzp27IyLiwuvvPIqI0YMwWw24+n5DH37DkixbLNmzTl7NpjAwD6YTGa6d+9FpUqVmTBhCv/73/ssX74Ug6EAkya9n+4ZqerVa7J48ULmz/+IJ598MsX8zp278eefV3nlld4kJSXRrl1HfH3rZTGjIiIi9iE2OoKzR7ZzKfgQVWo0oNPAUexYs5TQq39Qt0V3DAXcbB2iiGSRk9lsNts6iKy6eTMakyl52CEhl/HwqGyTeAwGZ4xGU55u85dfjrB48UI+/nhhnm43PXmdB1u+5ulxdy9CWFiUrcOwOeXhPuXCQnmwUB4sHDUPZw79SNzdKKrVb2kd7fD6tXCObv+eWyGXHXa0Q0c9HlKjXFjkpzw4OzvxxBOF05yvM1Qi8tDyYrAXebTomJAH6Xiw+OHCVjZG/38eDh5IPtMVyjxu5PiP/8O74Yt2nQcRe6MzVDnAFmeo8iOdobLIT7+o2JLycJ9yYRE4/ScWj37B1mHYnI4HC+XBQnmwUB7uUy4s8lMeMjpDpUEpREREREREskkFlYiIiIiISDapoBKRXLF8y1lbhyAi+ZjeI0TEXmhQChHJFd9s/Y1WvuVtHYbYyNo9F1i391KK6YHTf0r23L9xFTo3qZpHUUl+ovcIEbEXKqhERCTHdW5SNUWhpEEpRETEHumSv1wyfPhg+vbtwYABvRkwoDenT58CYMKEMXTv3olPP/3Y2vbrr5ewf/9eW4WaKXv37uHbb78GYO3alaxduxKA557Tn/SKiIiIiOPSGapcYDabuXr1CitXrsdguJ/iP/74nZiYaL7/Poj+/V+mb98BmExJBAefpm/fAbYLOBPOng22Pu7cuZsNIxERERERyT9UUOWCK1cuA/DWW8OIiIjA378zL730MgUKFCAhIYH4+DiMRiMuLi588cVn9Os3IM11bdy4nn379nD79m1u3gynceMmDBs2EicnJ776agk7dvxIUpKJhg39+Oc/RxAScp233x5OsWLFcXNzY8aMD5g9ewYnThzHYDAwYMAgWrRozZkzp/nww9nEx8dRrFhxRo0aQ7ly5Rk27DVq1KjJr78e586d27z55ig8PMoSFLQaAA+PsoSEXAdg4MDB1jjv3r3LnDn/5dy5c5hMJvr06U+rVi/mXpIlX9H9MiKSHr1HiIg9U0GVC6KiIqlbtz4jR76L0Whk+PDXqFSpMvXr++Hp+TQDB/ajc+du3L59i4iIO1SvXjPd9Z048StLliynSJGiDB8+mN27d+Dm9hi//XaGzz5bipOTE5Mn/4etWzfh7e3DlSuX+f77jyhbthzLly8lNjaWZctWcvv2Ld54YyhNmzZn+vQpzJjxAR4eHhw8uJ8ZM6Yyd+48ABITjSxY8AU//7ybzz6bz+LFX9OpU1cA2rf35/PPF6SI8csvP8fLqzpjxkwkJiaaIUMCqVGjFuXLV8j5BEu+o/tlJDN6tfaydQhiI3qPEBF75nAFVXjQGm6tD8qwXcmOnSjVqUu2tlGrlje1anlbn3fo0In9+/dSv74fI0a8bZ0+ZcoEXnnlVVav/p6ff96Ft7cPAwYMSrG+555rSsmSTwDQokVrjh49jKurG8HBpxg4sB8A8fFxlCnjgbe3DyVKlKRs2XIAHD/+C/7+XXB2duaJJ0rx9dcruHDhHNeu/cno0W9ZtxETE2N93LDhPwCoWvUpoqIiM7XPR44cIj4+jvX/n9u4uDguXryggspBxB/fiEvpJzGUq55mG+O1MySFXsTNp10eRia2ktox0btNtWT/eq9jwnHoPcIitTws33I22WiHjpAHEXvjcAVVqU5dsl0oZdavvx4nMTGBevUaAJZ7qh68lwrg7NkzFC5chBIlSrBmzfcsXfodb789gitXLlOpUuVkbV1c7i9rNptwcTFgMiXRo0cvevbsC0BUVBQuLi5ERNzBzc3tb8s6WZ//+edVkpJMlCtXniVLlgOQlJTE7du3rG1cXV0BcHJywmw2Z2qfTaYkJk6cgqen5RfoW7duUrRosUwtK48+l9JPErdtHo+1HJrqFybjtTPW+eIYdEzIg3Q8WKSWhweHj3eUPIjYG43ylwuio6OYN28u8fHx3L0bw6ZNP9C0afNkbb78chEBAQMxmSwFi5OTE05OTiQkJKRY38GD+4iOjiY+Pp4ff9yCn18jfH3rs2XLRu7evYvRaOTf/36bnTu3p1jWx6cOP/30I2azmdu3bzFs2GuULVuOyMhIfv31GAA//LCOiRPHprtPLi4uJCUlpTnf17c+q1dbRv4LDw8nIKAXN26EpJ8osRuGctV5rOVQ4rbNw3jtTLJ5D35BSO/XabEvOibkQToeLJQHEfvkcGeo8kLjxk0IDj5FYGAfkpJMdO3aPdklgPv3/4yXV3VKlCgBQIMGfnTv7o+3tw+enk+nWF/x4iUYNWoEd+7coU2bdtZL8s6d+53XXrOMFNiwYSPatu1gHTDini5dujNnzv8YMKAXACNHjqJw4cJMnjyduXNnkpCQQKFCjzNu3Hvp7pOPjy9Tp06kZMmSqc4PDHyV2bNn0K9fD0wmE0OHjtDlfg7mwS8Kj7UcSq/WXvqC4OD+fkzg3kDHhAPTe4RFin6BiimRR52TObPXdOUjN29GW8/s3BMSchkPj8ppLJG7DAZnjEZTrqx748b1HDt2lLFjJ+bK+nNSbuYhNbZ8zdPj7l4k2X0ijubeF4Ni9doQcWSLviCgY0LHRHI6HnQ8wP08bLr9JG1LXHTYPNzj6P3iQcqFRX7Kg7OzE088UTjN+TpDJSI5ylCuOgVqNOfOzytx9fV36C8IYqFjQh7kyMfD34ePb1vwSV4seIJNt73ZtPQ6YLnKRMPHizxaVFDlc+3adaRdu462DkMk04zXzpAYvIPiz3Uj4sgWXMpVd6gvTJKSjgl5kCMfDw8OH285Q7WGTbe9aVviIl26t3WYPIjYGw1KISI55sH7AEo265XmzdfiOHRMyIN0PFg8mIdNsT4OmwcRe6GCSkRyRGo3Vac3opXYPx0T8iAdDxbKg4j9UUElIg8tvRGq9EXBMemYkAfpeLBQHkTskwoqEXloSaHpj1B174tCUujFPI5MbCUp9CJnPQI4GFyAmOj4ZPNiouM5GFyAsx4BOiYchN4jLFLLQ6/WXtbHjpIHEXujQSlywfXr1+jVqytVqiQfoadjx8689FKPVJfZu3cPV69epmfPvqxda/mD3M6duz1UHA+uM7OGDXuNwMDX8PWtl6n2d+/eZdq0SVy5cgmAfv1eoWXLNinarVq1gvXr12I2m3FycuLll3vTtm0HALp168hjjz2GwVDA2v6ZZ7wYM2ZCpuMW23LzaZdhG4MD3XgulmOiRnQCxw5e4btFR/CqVYaGTapycM8Ffjt1A6/aZajRyBu3wq62DlXygN4jLFLLQ+821ZINDe0IeRCxNyqockmpUu4sWbI80+3Png22Pn7YQiq1deaWr79eQpkyHkyePJ3IyDv069cTX996lCz5hLXN6dOn2LBhLQsWLMbN7TFu377FwIH98PR8hqeffgaA//1vLmXLlsv1eEUk7xQq7ErjFp7UaViJYwev8NkHe/CuX56eg+pTSIWUiIjYCYcsqI4duMLd6AR8/CryeGE36/SY6HiOH7hKocKu1PGrlCvbNhqNTJv2HhcunAegS5fu1K79LEFBqwHw8ChLSIjlfygGDhyMv38bmjRpRnDwKUqWLEX79v6sXPktYWGhjBkzgTp16nLs2FEWLpxHfHwcUVHRjBgxkgoVKiVbZ/PmLZk9ewYXLpzHZDLRp09/WrV6kYSEBGbMmMzZs2fw8ChHRMSdLO1PnTq+VKxYBYCSJUtStGhRbt68maygunUrHLMZ4uLicHN7jBIlSjJlygxKlCjxsOkUkUfAvcLqxOG/aNzC09bhiIiI5CiHLKi8ankkuwzlmVpl+P3UDetlKF61PB56G+HhYQwY0DvZtPHjJxEZGUFkZCRffLGc8PAw5s//CH//LnTq1BWA9u39+fzzBdZlbt26iZ9fI0aNGsPw4YPZvXsH8+YtYtOmDaxY8Q116tRl1arvGD16PJUrV+Ho0cPMnTuTpUu/S7bO+fM/wsurOuPGvUdMTDRDhgRSo0Ytdu3aAcCyZSu5evUKAQG9srSf9ev7WR9v27aVxMREnnwy+aWOfn6N2bhxPZ06vUitWt7UqVOXF19sT6lS7tY2o0a9keySv+7de9K+vX+WYhERkfzLlj9miojkJocsqP5+GcrKJb/k+GUoaV3yFxFxhytXLvPWW8Pw82vM66+/keG6/PwaA5YzTd7ePgCUKeNBVFQkAOPHT2bfvj3s2LGN06dPEhsbm2IdR44cIj4+jh9+WAdYzhZdvHiB48eP4u9vKbwqVqxE7dre2drfn37axty5M5k580MMhuSHVYECBZg2bRZ//nmVQ4cOcODAPr755ivmzJlPrVq1AV3yZw/0ZUlE0vP3HzP/fk9dTvyYKSJiCw5ZUN1ji8tQihUrzldfreDw4YPs37+XwMC+fPXVinSXKVDg/pkbFxeXFPNff/1VfH3rUqdOXerWrc97741L0cZkSmL8+Ml4eVUDLGe+ihYtxrp1awBzuutftOhTfv55NwCDBg3mueeaJZu/cuW3LF/+FXPnfkKVKk+lWH7Tpg24u5emXr0GVKhQka5du7NgwSds2bLRWlDJo09flkQkPbqnTkTslYZNz2M//7yLyZP/Q6NGz/Hmm+9QsGBBQkNv4OLiQlJSUpbXFxkZwdWrlxk4cAh+fo3Zs2cXJpMJINk6fX3rW0cPDA8PJyCgFzduhFCvXgO2bt2MyWQiJOQ6J0+eSLGNQYOGsGTJcpYsWZ6imNq9eyffffcN8+d/jqfn06nGaDKZWLDgY+7csdyflZiYyKVLF3jmGa9U28uj6d6XpZ6D6oMTfPbBHnCCnoPq07iFp74wCU1bp/4eIY7l3nsFoPcGEbELmTpDtX79eubPn4/RaCQgIIA+ffokm3/mzBnGjh1LTEwM9erV47333sNgMHD06FGmTZtGYmIixYsX5/3336d8+fIcOnSI4cOH4+Fh+cW6Ro0aTJs2Lef3zoZSu4fKx6cOw4aNZOfOn+jXrweurq60adOOp57yJCoqkqlTJ1KyZMksbado0WJ06NCJfv16YDAY8PWtT1xcHLGxsfj4+FrXGRj4KrNmzaBfvx6YTCaGDh1B+fIV6Nq1OxcvnqdPn254eJSlatWUZ5jS8/nnC0hIiONf/3oLJycwm2H06HFUq1bD2qZ9e38iIu7wz38G4uxsqeFbtGhNhw6drG3+fg/VY489xqefLs5SLJI/aAACScvzbbySDQ8tIiJiD5zMZrM5vQY3btygV69erF69GldXV3r27Mns2bPx9Lz/RalDhw5MmTIFHx8fxowZQ61atejduzcvvPAC8+bNo1q1aqxcuZLt27czf/58Fi9eTGJiIoMHD85W0DdvRmMyJQ87JOQyHh6Vs7W++dN38c/RzTJumAaDwRmj0ZTt5e1FXufhYV7z3OTuXkRfGnnECxZGAAAP5klEQVT4fmVPdExYKA8WjpiH8KA13FoflGzads9XaHHui2TTSnbsRKlOXfIytDyVWh5SY+95SI0j9ou0KBcW+SkPzs5OPPFE4TTnZ3iGat++ffj5+VG8eHEA2rRpw+bNmxk2bBgAf/31F3Fxcfj4WAZL6Nq1Kx9++CHdunXjjTfeoFo1yz07Xl5efP311wCcPHmS8PBwNmzYQPny5ZkwYQJly5Z9uD19CPUa578v5SKPklS/JHi+wu+DBiSb5IhfEhxVasfE76m00zHhGEp16pLidd4+fRfPLFpim4BsJLU85KcvjSKSPRkWVKGhobi73x/eunTp0pw4cSLN+e7u7ty4cQNXV1c6dbJc0mUymfj4449p2bIlAEWKFKFt27a0bt2ab775hpEjR/Ltt99mOujUKsTQUGcMhuzdEvaP5lUzbpSB7G7b3uRlHpydnXF3L5Jn28uK/BpXbnEf1B8G9U82bfvbG2gctMpGEeU/OibkQY52PKSmaeunHTIPe386R3RUPI2aP0WRoo8BluMhKjKOfTvOU7iIG41fcMzLpR3xeEiLcmHxqOQhw4LKZDLh5ORkfW42m5M9z2h+QkICo0ePxmg0Wi/xmzRpknV+r169mDVrFlFRURQpkrmkpXbJn8lkstlld7rkzyKv82AymfLlr3r6tdGiaeunlYf/p2PCQnmwUB4sHPWeugpPluDYwSvMm7Ez1RFRq3l7OGRe1C/uUy4s8lMeMrrkL8PTCR4eHoSFhVmfh4WFUbp06TTnh4eHW+fHxMQwaNAgjEYj8+fPp0CBAphMJubPn59iRLvUhusWkUfX8200iqOIyN9pRFQR+5NhQdWoUSP279/PrVu3iI2NZevWrTRt2tQ6v3z58ri5uXH06FEAgoKCrPNHjRpF5cqVmTNnDq6uljcIZ2dnfvzxR7Zs2QLA2rVrefbZZylUqFCO75yIiIhIfqTh40XsR4aX/JUpU4aRI0fSv39/EhMT6datG97e3rz66quMGDGC2rVrM3PmTMaNG0d0dDQ1a9akf//+BAcHs337djw9PenSxXIDZunSpfnss8+YMWMG48eP55NPPqFkyZL897//zfUdFRERERERyWmZ+h+qjh070rFjx2TTPvvsM+vje8OiP6hGjRr89ttvqa7v6aefztIgFLnt1L6N1GrUztZhiIiIiIjIIyZTBZW9O31gc44WVNevX6NXr65UqZJ89MCOHTvz0ks9Ul1m7949XL16mZ49+7J2raU47dy520PF8eA6M2vYsNcIDHwNX996Wd7er78eZ86cmSQmGilWrBj//vd/8PBIPhx+VFQUs2fP4Pz5PwAoVao0I0eOomLFSvzyyxH+9a+RlC9fMdkyr7zyKs2aNc9yPCIiIiIiuU0FVS4pVcqdJUuWZ7r92bPB1scPW0ilts68MGHCWKZPn42n59Ns2BDEnDn/Y/r02cnaLFjwMVWrPsWECVMA+PHHzUyY8G8WL14GgJdXdT7+eGGexi0iImIrTVs/besQROQhqaDKY0ajkWnT3uPChfMAdOnSndq1nyUoaDUAHh5lCQm5DsDAgYPx929DkybNCA4+RcmSpWjf3p+VK78lLCyUMWMmUKdOXY4dO8rChfOIj48jKiqaESNGUqFCpWTrbN68JbNnz+DChfOYTCb69OlPq1YvkpCQwIwZkzl79gweHuWIiLiTZuy//HKETZs2MHbsxBTzEhISGDx4KJ6elg8GT8+nWbXquxTtbt68SYkSJTGZTDg7O9OiRWsKFtSAJCIi4pgcdfh4EXuigiqXhIeHMWBA72TTxo+fRGRkBJGRkXzxxXLCw8OYP/8j/P270KlTVwDat/fn888XWJe5desmfn6NGDVqDMOHD2b37h3Mm7eITZs2sGLFN9SpU5dVq75j9OjxVK5chaNHDzN37kyWLv0u2Trnz/8IL6/qjBv3HjEx0QwZEkiNGrXYtWsHAMuWreTq1SsEBPTK1v66urrStm17jEYTJpOJxYsX0qTJ8ynaBQQMZMyYd1iz5nt8fetTv35DWrdua53/229nUuRt7tx5FCtWPFtxiYiIiIjkJocuqGKjIzh7ZDsAF08f5MmaDXNs3Wld8hcRcYcrVy7z1lvD8PNrzOuvv5Hhuvz8GgOWM03e3j4AlCnjQVRUJADjx09m37497NixjdOnTxIbG5tiHUeOHCI+Po4fflgHQFxcHBcvXuD48aP4+1sKr4oVK1G7tneKZXft2sEXX3xGbOxdIiMjGTCgN88848WYMRNStE1MTGTKlAkYjUn07x+YYn61atVZsSKIkyd/5ciRQ3z77dcEBa3m008XA7rkT0REREQeLQ5ZUN0rpC4FH6JKjQY83+11fvlpJaFX/6Bui+4YCrjl2raLFSvOV1+t4PDhg+zfv5fAwL589dWKdJcpUKCA9XFqf4D8+uuv4utblzp16lK3bn3ee29cijYmUxLjx0/Gy6saYDnzVbRoMdatWwOY011/s2bNadasebqX/AHcvXuXd955k6JFizF9+iwMhuSHl9lsZtas6Qwf/hZ16ljifeWVV+nZswt//JH6iJAiIiIiIvlZhn/sa48uBR8C4MWAMdR5vitlKnnRqs87APy4bCYR4ddzbds//7yLyZP/Q6NGz/Hmm+9QsGBBQkNv4OLiQlJSUpbXFxkZwdWrlxk4cAh+fo3Zs2cXJpMJINk6fX3rW0cPDA8PJyCgFzduhFCvXgO2bt2MyWQiJOQ6J0+eyPa+TZw4jvLlKzJp0jTrHzk/yMnJiUuXLvLNN19ZY7x27S+SkpIoX75CtrcrIiIiImIrDnmGqnqDVimmGQq40fDFvlw8fZBrF09TrFTZVJbMvNTuofLxqcOwYSPZufMn+vXrgaurK23atOOppzyJiopk6tSJlCxZMkvbKVq0GB06dKJfvx4YDAZ8fesTFxdHbGwsPj6+1nUGBr7KrFkz6NevByaTiaFDR1C+fAW6du3OxYvn6dOnGx4eZala9ak0t+XrWy/N4dR///0su3fvpEqVqgQGWoZpL1WqFDNnfpis3cSJ7/PRR7Po3t2fggUL8vjjhZkwYQpFixYDUr+HqkWL1vTrNyBLeRERERERyQtOZrPZnHGz/OXmzWhMpuRhh4RcxsOjsk3iMRicMRpNNtl2fpLXebDla54ed/ciGrEJ5eFByoWF8mChPFgoDxbKg4XycJ9yYZGf8uDs7MQTTxROe34exiIiIiIiImJXVFCJiIiIiIhkkwoqERERERGRbLKrguoRvB1MsslsNgFOtg5DRERERByc3RRUBoMrMTGRKqrsnNlsxmhM5M6dcFxdH7N1OCIiIiLi4Oxm2PQSJdy5fTuM6Og7eb5tZ2dn6/8qObK8yoOzswsFCxamcOFiub4tEREREZH02E1B5eJioNRD/ndUduWnYR1tSXkQEREREUdjN5f8iYiIiIiI5DUVVCIiIiIiItn0SF7y5+yc/0Z3y48x2YLyYKE8WCgP9ykXFsqDhfJgoTxYKA8WysN9yoVFfslDRnE4mTUsnoiIiIiISLbokj8REREREZFsUkElIiIiIiKSTSqoREREREREskkFlYiIiIiISDapoBIREREREckmFVQiIiIiIiLZpIJKREREREQkm1RQiYiIiIiIZJMKKhERERERkWxSQZUJ69evp127drRu3Zply5YBsG/fPjp27Ejr1q354IMPrG3PnDlD165dadOmDWPHjsVoNNoq7ByXlTzc8+6777J69eq8DjVXZSUP27Zto1OnTvj7+zN06FAiIiJsFXauSC0XAImJiQQEBHDw4EHrNEfrG5B6Hu5xlL4BqefB3vtGdHQ0HTp04M8//wQc8zMDMp+He+yxX0Dm8+Bo/QIc7/Pinszm4h5H6RvwiH5mmCVdISEh5ubNm5tv375tjomJMXfs2NH8+++/m5s1a2a+cuWKOTEx0RwYGGjeuXOn2Ww2m9u3b28+duyY2Ww2m//973+bly1bZsvwc0xW8xASEmIePHiw2dvb27xq1SobR59zspKHqKgoc+PGjc0hISFms9lsnjNnjnny5Mk23oOck1ou/vjjD/P58+fNL7/8srl27drmAwcOWNs7Ut9ILw+O1DfSyoO9943jx4+bO3ToYK5Zs6b56tWr5tjYWIf7zDCbs5YHe+0XZnPm8+Bo/cJsNjvc58U9WcmFI/UNszn1PDwKfUNnqDKwb98+/Pz8KF68OIUKFaJNmzZs2bKFypUrU7FiRQwGAx07dmTz5s389ddfxMXF4ePjA0DXrl3ZvHmzjfcgZ2QlD2D5pbpFixa0bdvWxpHnrKzkITExkQkTJlCmTBkAvLy8uH79uo33IOeklovNmzezcuVKBg0axLPPPmtt62h9I608gGP1jbTyYO99Y8WKFUyYMIHSpUsDcOLECYf7zIDM5wHst19A5vPgaP0CcLjPi3symwtwrL4BqefhUegbKqgyEBoairu7u/V56dKlWbduXYppN27cSNHW3d2dGzdu5Gm8uSUreQAYNGgQ3bt3z/M4c1tW8lCiRAlatWoFQFxcHAsXLqRly5Z5HnNuSS0XN27c4N13302xn47WN9LKAzhW30grD/beN6ZOnUq9evWsz9PKjT33C8h8HsB++wVkPg+O1i8Ah/u8uCezuQDH6huQeh4ehb6hgioDJpMJJycn63Oz2QyQYpqTk1OqbR98/ijLSh7sWXbyEBUVxWuvvUa1atXo0qVL3gWby7JyvDta37CXfcuK7OTBXvvG36WVG0c7dhxtf9OSUR4cpV+kRceJpCU/9w0VVBnw8PAgLCzM+jwsLAx/f/8U00qXLp2ibXh4eLLTmI+yrOTBnmU1D6GhofTu3RsvLy+mTp2a5/HmptRykdbr72h9w172LSuymgd77ht/l1Zu7LlfpEZ9xSK9PDhSv0iLo/ULyZz83jdUUGWgUaNG7N+/n1u3bhEbG8vWrVtp2LAhFy9e5PLlyyQlJbFhwwaaNm1K+fLlcXNz4+jRowAEBQXRtGlTG+9BzshKHuxZVvKQlJTEkCFDaNu2LWPHjrW7X9hSy0Var7+j9Q172besyEoe7L1v/N2zzz7rcJ8ZqUkrD44mrTw4Wr9Ii6P1C8nYo9A3DLYOIL8rU6YMI0eOpH///iQmJtKtWzfq16/P9OnTGT58OPHx8TRr1owXX3wRgJkzZzJu3Diio6OpWbMm/fv3t/Ee5Iys5sFeZSUP27ZtIzg4mKSkJLZs2QJArVq18uUvK9mRWi68vb3TbO9IfSO9PNirrOThp59+suu+8Xdubm4O95mRmvTy4EjSyoO9f2ZkhSP1C8nYo/CZ4WS+dxOIiIiIiIiIZIku+RMREREREckmFVQiIiIiIiLZpIJKREREREQkm1RQiYiIiIiIZJMKKhERERERkWxSQSUiIiIiIpJNKqhERERERESySQWViIiIiIhINv0fftHG+HRtz2YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "\n",
    "sns.set()\n",
    "\n",
    "ind = np.arange(12)\n",
    "labels = ['000', '001', '002', '010', '011', '012', '100', '101', '102', '110', '111', '112']\n",
    "\n",
    "x = np.arange(len(labels))\n",
    "fig, ax = plt.subplots(figsize=(12, 8))\n",
    "ax.plot(x, theta_hat, label='Estimated', linestyle='', marker='+', ms=10)\n",
    "ax.plot(x, true_distribution, label='Actual', linestyle='', marker='x', ms=10)\n",
    "ax.plot(x, theta_p95, label='95% percentile', linestyle='', marker='_', ms=10)\n",
    "ax.plot(x, theta_p05, label='5% percentile', linestyle='', marker='_', ms=10)\n",
    "ax.plot(x, theta_hat - 2*se_theta_hat, label='Estimated - 2 SE', linestyle='', marker='1', ms=10)\n",
    "ax.plot(x, theta_hat + 2*se_theta_hat, label='Estimated + 2 SE', linestyle='', marker='2', ms=10)\n",
    "\n",
    "ax.set_title('Estimated vs actual parameters (with confidence + SE bounds)')\n",
    "ax.set_xticks(x)\n",
    "ax.set_xticklabels(labels)\n",
    "ax.legend()\n",
    "\n",
    "fig.tight_layout()\n",
    "\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 20.8.4**. Let $V = (X, Y, Z)$ have the joint distribution\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "X &\\sim \\text{Bernoulli}\\left(\\frac{1}{2}\\right) \\\\\n",
    "Y | X = x &\\sim \\text{Bernoulli}\\left(\\frac{e^{4x - 2}}{1 + e^{4x - 2}}\\right) \\\\\n",
    "Z | X = x, Y = y &\\sim \\text{Bernoulli}\\left(\\frac{e^{2(x+y)-2}}{1 + e^{2(x+y)-2}}\\right)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "**(a)** Find an expression for $\\mathbb{P}(Z = z | Y = y)$.  In particular, find $\\mathbb{P}(Z = 1 | Y = 1)$.\n",
    "\n",
    "**(b)** Write a program to simulate the model.  Conduct a simulation and compute $\\mathbb{P}(Z = 1 | Y = 1)$ empirically.  Plot this as a function of the simulation size $N$.  It should converge to the theoretical value you computed in (a).\n",
    "\n",
    "**(c)** Write down an expression for $\\mathbb{P}(Z = 1 | Y := y)$.  In particular, find $\\mathbb{P}(Z = 1 | Y := 1)$.\n",
    "\n",
    "**(d)** Modify your program to simulate the intervention \"set $Y = 1$\".  Conduct a simulation and compute $\\mathbb{P}(Z = 1 | Y := 1)$ empirically.  Plot this as a function of the simulation size $N$.  It should converge to the theoretical value you computed in (c)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(a)**\n",
    "\n",
    "We have:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbb{P}(Z = z | Y = y) &= \\frac{\\mathbb{P}(Y = y, Z = z)}{\\mathbb{P}(Y = y)} = \\frac{p(y, z)}{p(y)} \\\\\n",
    "&= \\frac{\\sum_x p(x, y, z)}{p(y)} = \\frac{\\sum_x p(x) p(y | x) p(z | x, y)}{p(y)} \\\\\n",
    "&= \\sum_x p(z | x, y) \\frac{p(y | x) p(x)}{p(y)} = \\sum_x p(z | x, y) \\frac{p(x, y)}{p(y)} \\\\\n",
    "&= \\sum_x p(z | x, y) p(x | y)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Let $\\alpha = \\frac{e^{-2}}{1 + e^{-2}}$.  Note that $\\frac{e^{-2}}{1 + e^{-2}} + \\frac{e^{2}}{1 + e^{2}} = 1$, so $1 - \\alpha = \\frac{e^{2}}{1 + e^{2}}$.\n",
    "\n",
    "\n",
    "Looking explicitly at each potential outcome for the joint distribution of $X$ and $Y$, we get:\n",
    "\n",
    "$$\n",
    "\\begin{array}{c | cc | c}\n",
    "p(x, y) & Y = 0 & Y = 1 & \\\\\n",
    "\\hline\n",
    "X = 0 & (1 - \\alpha) / 2 & \\alpha / 2 & 1/2 \\\\\n",
    "X = 1 & \\alpha / 2 & (1 - \\alpha) / 2 & 1/2 \\\\\n",
    "\\hline\n",
    "      & 1/2 & 1/2 & 1\n",
    "\\end{array}\n",
    "$$\n",
    "\n",
    "Therefore,\n",
    "\n",
    "$$ X | Y = 0 \\sim \\text{Bernoulli}\\left( \\alpha \\right) \n",
    "\\quad \\text{and} \\quad\n",
    "X | Y = 1 \\sim \\text{Bernoulli}\\left( 1 - \\alpha \\right) \n",
    "$$\n",
    "\n",
    "or, more generally,\n",
    "\n",
    "$$\n",
    "X | Y = y \\sim \\text{Bernoulli}\\left( \\frac{e^{4y - 2}}{1 + e^{4y - 2}} \\right) \n",
    "$$\n",
    "\n",
    "Then, for $Z = 1$ and $y \\in \\{ 0, 1 \\}$,\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbb{P}(Z = 1 | Y = y) &= \\sum_x p(z = 1 | x, y) p(x | y) \\\\\n",
    "&= p(z = 1 | x = 0, y) p(x = 0 | y) + p(z = 1 | x = 1, y) p(x = 1 | y) \\\\\n",
    "&= \\frac{e^{2y - 2}}{1 + e^{2y - 2}} \\left( 1 - \\frac{e^{4y - 2}}{1 + e^{4y - 2}} \\right) + \\frac{e^{4y - 2}}{1 + e^{4y - 2}} \\frac{e^{4y - 2}}{1 + e^{4y - 2}}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "More explicitly,\n",
    "\n",
    "$$ \n",
    "\\begin{align}\n",
    "\\mathbb{P}(Z = 1 | Y = 0) &= \\alpha (1 - \\alpha) + \\alpha^2 \\\\\n",
    "&= \\alpha  &\\approx 0.1192 \\\\\n",
    "\\mathbb{P}(Z = 0 | Y = 0) &= 1 - \\mathbb{P}(Z = 1 | Y = 0) \\\\\n",
    "&= 1 - \\alpha &\\approx 0.8808\\\\\n",
    "\\mathbb{P}(Z = 1 | Y = 1) &= \\frac{1}{2} \\alpha + (1 - \\alpha)^2 &\\approx 0.8354\\\\\n",
    "\\mathbb{P}(Z = 0 | Y = 1) &= 1 - \\mathbb{P}(Z = 1 | Y = 1) \\\\\n",
    "&= 1 - \\left(\\frac{1}{2} \\alpha + (1 - \\alpha)^2 \\right) &\\approx 0.1646\\\\\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(b)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def generate_samples(n):\n",
    "    seeds = np.random.uniform(low=0, high=1, size=(n, 3))\n",
    "    \n",
    "    x = np.where(seeds[:, 0] < 1/2, 1, 0)\n",
    "    y = np.where(seeds[:, 1] < np.exp(4*x - 2) / (1 + np.exp(4*x - 2)), 1, 0)\n",
    "    z = np.where(seeds[:, 2] < np.exp(2*(x + y) - 2) / (1 + np.exp(2*(x + y) - 2)), 1, 0)\n",
    "    \n",
    "    result = np.zeros((n, 3), dtype=int)\n",
    "    result[:, 0] = x\n",
    "    result[:, 1] = y\n",
    "    result[:, 2] = z\n",
    "    \n",
    "    return result\n",
    "\n",
    "def estimate_z_given_y(X, z, y):\n",
    "    # Just estimate 0.5 if there are no valid samples\n",
    "    if np.sum((X[:, 1] == y)) == 0:\n",
    "        return 0.5\n",
    "    return np.sum((X[:, 1] == y) & (X[:, 2] == z)) / np.sum((X[:, 1] == y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate data\n",
    "X = generate_samples(10**6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Estimate with various segments of the data\n",
    "n_values = np.floor(np.logspace(0, 6, base=10)).astype(int)\n",
    "z_estimates = [estimate_z_given_y(X[:k], 1, 1) for k in n_values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtIAAAH4CAYAAABntQpnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dfZidZ10n8O9MQwLYhsIYDYVCMcqNVKXy5kaoUaisvCiawuWCuryI1QVFBOuisFhfqqxVedEW0YKUXasXlrC+EcTaErFEoQiogd4roXShbdY6hQ0tNSGd2T/OmTKZZiZz7s455zmTz+e6eiXnec557t+ZX9J85577uc/U/Px8AACAwUyPuwAAAJhEgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAECDDeMuAGA1SinzSf45yZ1LTn1vrfXTa3D9X0zyyVrr249x7qNJvr3W+vmG674vyW/XWq9Ygxpfk+RjtdY/WanehuveL8m7aq1PuqfXWsVYb0ryXUkur7W+atHxH0vyY0ufnuQPaq0vWnKN5yc5o9Z6wZLjFyd5TJIn1Frv7B87KcnfJLm61vrqe1j7qf1rvbDWeu09uRawPgjSwCT5jlrrvw3jwrXW16xw7qxhjNngSUk+nqxcb4P7J3n8Gl5vJT+a5CG11s8uPlhr/Z0kv7PwuJTynCRvSHLhANd+RZIPJfnZJL/cP/az6X3z9fP3oOaUUp6W5HVJzrgn1wHWF0EamHillG9P8qtJ/k96s5i3J3ltkpf2H7+z1vpT/ef99yQ3JHlEkjuSPL/W+olSytuS/HOt9ddLKYeS/EmSRyX5gfTC2ZZa67+VUn42yfOSHEnyL0me3//9m5J8XZKZJF9I8txaa12h5vulFxS/Mcm9kvx1kvNrrUdKKb+Q5PuSHE4y2x9jZ5LHJrmolHJnkmcuqvffk/xmknOSnJzkgiTP7l/7piTfXWu9vZTywvSC7MYkD0jy2lrrm5L8fpL79GfeH5Pk4f3aZpKclOSNtda3llJO7j/365LMJflwkh+ttc4teW9nJvnt/uvnk/xGrfXtpZT3J5lKsruU8uJa6/uX+dp8S5I39+u+frmv4VK11n8vpTw3yTWllD/vj/XiJI9bmKFeMs4Hktx3yeFraq0vOcblX5rkB5P88WrrAdY/a6SBSXJ1KeWji/5716Jzj0svGJ6V5GB6M5FPT/LoJC8ppZzWf95jk/xWrfWb0guF/+MY42xM8me11rL4R/illO9JL9Rur7V+Q5Lrk/x4kqcm+XytdXut9eHpBe8fP857eV2SD9daH5Pkm5N8ZZKXl1JOT/Ky9MLfY5O8N8m31FovTnJtemH7XUuutSnJgVrr45NcluTS/jUemeR+SZ7ZD8E/kuRptdZvTvL9SX6t//oXJLmj/7WbSnJFklf2a9uR5KdLKf8hvXB/Sv95j+u/9msWF1JK2ZDkT/Plr/FTk/xKKWV7rfXs/tO+Y4UQ/aAk70ryilrrnuN8De+m1vpPSV7V/xq8NckP11pvXOa531prPWvJf8cK0am1flet9UOD1gOsb2akgUmy0tKO62utH+n/fn+S/1drPZzk30opB9ObgU16a4wXQtxbk1xcSpk5xvWOFfTOSfLHtdbPJUmt9eULJ0opnyql/ESSr03y7Un2Hue9PCPJ40spP9x/fJ/+r7+e5GNJ/qGUsjvJ7lrrXx/nWknyzv6v+5P800J4LKVcn+QBtdbbSinPSPL0UsrXJTkrvdnrpR6eZFuSt5ZSFo7dJ72w/570QvH7kvxVktfXWj95jNffu9a6K0lqrTeVUt6Z3rroFb8mpZT7JPlfSd5Ra/29VbznY6q1/lYp5VlJPl5r3b3CeIPMSAPcjSANrBeHljz+0jLPO7Lo91P9X+/2Y/8kty3z2vmFB/2bz05Nb9b1vPSWM1ye5NYkDztOvScleXat9ROLrjVfa50rpexIb+b8nCSvK6W8p9b6M8e53uL3f7f3Xkp5cHpB9neT/G16s87PWKau/7d4XXgp5av7x/69lLLwjcKTklxZSjmv1vpnS14/n6NNp7d85Xjekt5Slles4rnHc31631Qsq9b6rWswDnACs7QDONGcVUr5pv7vz0vygQF247gyyc5Syub+4wuSvDzJf0zytlrrW5LUJN+dXqBcyV8m+alSylQpZVN6yyF+vJTyqPR2J/lErfVX01sCsrCM4khWF0iP5bFJbknvJrz3ph+i+7taHElyUillql//HaWUH+yfP71fz2NKKf8lveUw7621/tf+e3j0knGuS/KlUsrO/utPS3JuejPYyyql/Fx6s+Tff6z1zABdZEYamCRX92+0W+znknxxgGscSHJhKeWMJP+a5IdW+8Ja67tLKY9M72a2JNmX3rrjRyX53f4yjan0Zn6/8TiXe2l6N/T9U3rh+Mokv1Zr/VIp5R1Jri2l3JbeDZEv7b/mT5P8aill42prXuS9SV6YXlCeS7InvWD9tUk+meSD/fdzdno3Mr6hlPIz/dr+W631moVtAJN8vJTyxfRu7nzj4kH69X9vkjeWUi5I79+ZX6y1Xn2c+n4pyY1J9ixaUpIkH6m1vqDh/QIM3dT8/NKfwAGsT/1dO367f6MgE2q5faQBRs3SDgAAaGBGGgAAGpiRBgCABoI0AAA0mNRdOzaltx3UzTn2/q8AALAWTkrywPQ+tfaozyyY1CD9uBz7U8cAAGAYzk7vA63uMqlB+uYk+dznbs/c3OhvlpyZOTmzs8f60DPGRU+6SV+6R0+6R0+6SV+6Z1w9mZ6eyv3v/xVJP38uNqlB+s4kmZubH0uQXhibbtGTbtKX7tGT7tGTbtKX7hlzT+62nNjNhgAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQYMO4CwBWb+++A9m1Z39mDx7KzOZN2bljW7afuXXcZQHACUmQhgmxd9+BXLb7uhw+MpckmT14KJftvi5JhGkAGANLO2BC7Nqz/64QveDwkbns2rN/TBUBwIlNkIYJMXvw0EDHAYDhEqRhQsxs3jTQcQBguARpmBA7d2zLxg1H/5XduGE6O3dsG1NFAHBic7MhTIiFGwrt2gEA3SBIw5i0bGW3/cytAwXnljFGscWebfwAWA8EaRiDUWxl1zJGV+sCgC6yRhrGYBRb2bWM0dW6AKCLBGkYg1FsZdcyRlfrAoAuEqRhDEaxlV3LGF2tCwC6SJCGMRjFVnYtY7S8Zu++Azn/kmvywtdelfMvuSZ79x1Y87oAoIvcbAhjMIqt7FrGGPQ1LTcOjmobPzuDADBsgjSMyaBb2Y1qjEFes9KNg8cL7MN87607lgjeAAzC0g6gWVdvHBx0Z5CF4L1Q90LwPt4yFQBObII00KyrNw4OGvBtyQdAC0EaaNbVGwcHDfhdnVkHoNuskQaadfXGwZ07th21RjpZOeDPbN50zNA87pl1ALpNkAbukS7eODhowB80eANAIkgDHTeKnUFGNbMOwPoiSAOdNqr1y4POrI9iuzxb8gF0myANdFoX1y+3LDfp4hgA3DN27QA6rYs7g4xiuzxb8gF0nxlpoNO6uH55FMtNWsewHARgdARpWCMCzPCM4uPUBzGK5SYtY1gOAjBalnbAGvAR0yeWUSw3aRnDchCA0TIjDWugdYs2JlPrcpNBfmrRMoZPaByuQX/q1PJTKj/ZgskiSMMaEGBOPC3b5bV8sMwgY7QuBzkRg1tLKB6kfy397urSnBP1zwishiANa6CLW7TRLaP4qcWgn9DY1eA2bC3ve9D+tfS75TXDniVv/YZg2DPxwv1kW0/9E6RhDfiIaY5nFD+1GHQ5SGu4n/R/BFve96D9a+n3oK8ZxSz5oF+rUczEjyrcv+/Dn8nb/nxfp5bydHGMUX5zduvBQ3lAx/6fI0jDGujiFm10y6h+ajHIcpCWsNfVWexB/jFved+D9q+l34O+ZhSz5IN+rUYxEz+qcP/299Qc+tKdq3pNl7+BGOYYXf3mbJTs2gFrZPuZW3PRi5+Qt77ySbnoxU/oxF9wuqOLHyyzUghcTsvOIHv3Hcj5l1yTF772qpx/yTVrvpvNoLvmtLzvQfvX0u9BXzOKWfJBv1ajmIlfy3C/nF179t8VolfzmtYxBnlNF8doqWkU/RslQRpgBLafuTXPe+oj7gogM5s35XlPfcRYv+FqCXutyw8G2Rpy0OA96D+0Le970P619HvQ1wwackfxDUTLGMN+H10M9+tljK5+czZKlnYAjEjXPlimZUnSsJcftPwYd9B/aFuXYg3av5Z+D/KaQe/NaLmXY9CvVcsYw34fo1hmc6KO0VLTKPo3SiddcMEF466hxalJXnbHHYczPz/6wb/iKzbli188PPqBWZaedJO+dM/Snpz+VSfnKY97SJ75xIflKY97SE7/qpNXfP0p992Yf/7UbO6c+/L/fDdumM5zznn4MV/7R3/9yWNe545Dd+aZT3zY3Y6/8YqP5bY7jhx17M65+dxw4GCe8riHHPNaf/uPN+WOQ3fe7fjM5k3LvmbQ9z1MrX9PTv+qkzNzv3vnhgMHc8ehOzOzeVOec87Dlw25gz5/8etW+7VqGWPY72PQP7MLr9l3/a2rfk3rGIO8potjtNQ0iv6ttampqdz3vhuT5A1JPr/4nBlpAFZt0BnKQWeTWn6MeyLvmjOKWfJBDXsmftDnt/wEYvuZW7P5lHuveteO1jEGeU0XxxjFT3cWj9HFXTum5scxpXvPnZHk+tnZ2zI3N/r6t2w5Jbfc8oWRj8vy9KSb9KV7Rt2TpUs1kl7IXW7t7/mXXLNs8L7oxU9YcZxJ3TXH35Nu0pfuGVdPpqenMjNzcpI8LMmnF58zIw3A0Ixije3COJMSnIH1Q5AGYKiG/WN4gHERpAHoFLPLwKSwjzQAADQQpAEAoIEgDQAADYa6RrqU8twkr05yrySvr7VevOT8o5O8OcnGJJ9J8oO11s/f7UIAANAxQwvSpZQHJbkwyWOSHErygVLK1bXWjy962huSvKbWuruU8htJfjq94A13M6p9Yid5P1oAYHSGubTjnCRX1VpvrbXenuSKJM9a8pyTkmzu//6+Se4YYj1MsIUPdVj4oIbZg4dy2e7rsnffgYkcBwCYfMMM0qcluXnR45uTPHjJc16e5PdKKTcn+c4kvzPEephgu/bsP+oDGpLk8JG57NqzfyLHAQAm3zDXSE8nWfz53VNJ7koopZT7JHlLknNqrR8spbw8yduTPH21A/Q/rnEstmw5ZWxjn4huPcZHBi8cX+jFWvRkNeMwGF+37tGT7tGTbtKX7ulaT4YZpD+b5OxFj7cmuWnR429Icket9YP9x29O8kuDDDA7e1vm5uaP/8Q1Nq7Pej+RPWDzpruWWyw9fsstX1iznhxvHAbj70r36En36Ek36Uv3jKsn09NTy07eDnNpx5VJnlxK2VJKuW+Sc5O8Z9H5TyY5vZRS+o+fmeRDQ6yHCbZzx7Zs3HD0H9eNG6azc8e2iRwHAJh8Q5uRrrXeWEp5VZKr09ve7tL+Eo53p7dTx7WllOcneUcpZSrJvyZ5wbDqYbIt7Jox7N00RjUOADD5pubnR780Yg2ckeR6SztYoCfdpC/doyfdoyfdpC/d04GlHQ9L8umjzo28GgAAWAcEaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQYMO4C4Bh2rvvQHbt2Z/Zg4cys3lTdu7Ylu1nbh13WQDAOiBIs27t3Xcgl+2+LoePzCVJZg8eymW7r0sSYRoAuMcs7WDd2rVn/10hesHhI3PZtWf/mCoCANYTQZp1a/bgoYGOAwAMQpBm3ZrZvGmg4wAAgxCkWbd27tiWjRuO/iO+ccN0du7YNqaKAID1xM2GrFsLNxTatQMAGAZBmnVt+5lbBWcAYCgs7QAAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANNoy7AE5se/cdyK49+zN78FBmNm/Kzh3bsv3MreMuCwDguARpxmbvvgO5bPd1OXxkLkkye/BQLtt9XZII0wBA51nawdjs2rP/rhC94PCRuezas39MFQEArJ4gzdjMHjw00HEAgC4RpBmbmc2bBjoOANAlgjRjs3PHtmzccPQfwY0bprNzx7YxVQQAsHpuNmRsFm4otGsHADCJBGnGavuZWwVnAGAiWdoBAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAabBjmxUspz03y6iT3SvL6WuvFS86XJG9Ocv8kB5L8p1rr54ZZEwAArIWhzUiXUh6U5MIkT0xyVpLzSimPXHR+KsmfJnltrfVRST6S5JXDqgcAANbSMJd2nJPkqlrrrbXW25NckeRZi84/Osnttdb39B//SpKLAwAAE2CYSztOS3Lzosc3J3n8osdfm+RAKeUtSb45ySeS/MQQ6wEAgDUzzCA9nWR+0eOpJHNLxv72JN9Wa722lPJLSX4zyfNXO8DMzMn3vMpGW7acMraxOTY96SZ96R496R496SZ96Z6u9WSYQfqzSc5e9HhrkpsWPT6Q5F9qrdf2H/9hess/Vm129rbMzc0f/4lrbMuWU3LLLV8Y+bgsT0+6SV+6R0+6R0+6SV+6Z1w9mZ6eWnbydphrpK9M8uRSypZSyn2TnJvkPYvOfyDJllLKo/qPvzvJh4dYDwAArJmhBela641JXpXk6iQfTXJ5rfWDpZR3l1IeW2u9I8n3Jfm9Usq+JE9K8oph1QMAAGtpqPtI11ovT3L5kmNPW/T7v8/RNyACAMBE8MmGAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA02DDuAibJRX/4D/nEDZ+/6/HXP/TUnP+cRw993L37DmTXnv2ZPXgoM5s3ZeeObdl+5taxXWfU1wYA6CIz0qu0NEQnySdu+Hwu+sN/GOq4e/cdyGW7r8vswUNJktmDh3LZ7uuyd9+BsVxn1NcGAOgqQXqVlobo4x1fK7v27M/hI3NHHTt8ZC679uwfy3VGfW0AgK5admlHKeXlK72w1vqba18OSy3M8q72+LCvM+prAwB01UprpL8pyblJ/jjJ1JJz80OriKPMbN50zEA6s3nTWK4z6msDAHTVskG61vr8UsrpSd5ba/2jEdbUSV//0FOPuYzj6x966lDH3bljWy7bfd1RSyc2bpjOzh3bxnKdUV8bAKCrjrdG+iVJnjCKQrru/Oc8+m6heRS7dmw/c2ue99RH3DW7O7N5U5731EcMvCPGWl1n1NcGAOiqqfn5iVylcUaS62dnb8vc3Ojr37LllNxyyxdGPi7L05Nu0pfu0ZPu0ZNu0pfuGVdPpqenMjNzcpI8LMmnjzo38moAAGAdEKQBAKCBIA0AAA0EaQAAaHDcIF1K2bv4VwAAYHUz0gufqnHvYRYCAACTxNIOAABoIEgDAEADQRoAABqsJki/v//r3wyzEAAAmCTHDdK11p9c/CsAAGBpBwAANBGkAQCgwbJBupQyNcpCAABgkqw0I31VKeWrR1YJAABMkJWC9IeSfKSU8p2jKgYAACbFskG61vozSb4/yZtKKReWUqynBgCAvg0rnay1vr+U8qgkv5/e7PQNi859z7CLAwCArloxSPf95yTfkeSSJJ8abjkAADAZlg3SpZTTkrwtyelJnlJr/cioigIAgK5bad3zPye5IcljhGgAADjaSks7fqTW+s6RVQIAABNkpV07hGgAAFiGLe0AAKCBIA0AAA0EaQAAaLDS9ndXJ5lf4bVTSeZrrU9a86oAAKDjVtq14/mjKgIAACbNskG61nrDcucAAOBEZ400AAA0EKQBAKBBU5AupUytdSEAADBJlg3SpZT/XUp5xjKnPzykegAAYCKsNCN9apK3llJ+5hjnzEgDAHBCWylI35jk25KcV0q5rJSycdG5lfaXvksp5bmllI+XUv6llPKSFZ739FLK9asrGQAAxm+lID1fa70uyeOTPDjJ+0opW/rnjjsjXUp5UJILkzwxyVnpBfJHHuN5X53k11dzTQAA6Irj3mxYa701yVOSfCTJtaWUs7K6GelzklxVa7211np7kiuSPOsYz7s0yS+svmQAABi/lT7Z8K4Z4lrrnUleUkr5xyR/ndUF6dOS3Lzo8c3pzW7fpZTy0iT/kOTvVlswAAB0wUpB+kVLD9Ra31xK+USS16zi2tM5OnBPJZlbeFBK+YYk5yZ5cnpLRwY2M3Nyy8vWxJYtp4xtbI5NT7pJX7pHT7pHT7pJX7qnaz1Z6SPCj7nFXa31b9JbtnE8n01y9qLHW5PctOjxs5M8MMm1STYmOa2U8v5a6+LXrGh29rbMza3qvsc1tWXLKbnlli+MfFyWpyfdpC/doyfdoyfdpC/dM66eTE9PLTt5u9KM9D11ZZIL+jco3p7e7PN5CydrrT+f5OeTpJRyRpL3DRKiAQBgnIb2EeG11huTvCrJ1Uk+muTyWusHSynvLqU8dljjAgDAKAxzRjq11suTXL7k2NOO8bxPJzljmLUAAMBaGtqMNAAArGeCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGiwYZgXL6U8N8mrk9wryetrrRcvOf/MJL+QZCrJ9UleUGv93DBrAgCAtTC0GelSyoOSXJjkiUnOSnJeKeWRi85vTvKmJE+vtT4qyT8muWBY9QAAwFoa5tKOc5JcVWu9tdZ6e5Irkjxr0fl7JXlJrfXG/uN/TPKQIdYDAABrZphLO05LcvOixzcnefzCg1rrbJJ3JUkp5T5JXpnkt4ZYDwAArJlhBunpJPOLHk8lmVv6pFLK/dIL1B+rtV42yAAzMyffowLviS1bThnb2BybnnSTvnSPnnSPnnSTvnRP13oyzCD92SRnL3q8NclNi59QSnlgkr9MclWSnxp0gNnZ2zI3N3/8J66xLVtOyS23fGHk47I8PekmfekePekePekmfemecfVkenpq2cnbYQbpK5NcUErZkuT2JOcmOW/hZCnlpCR/luQdtdZfHmIdAACw5oYWpGutN5ZSXpXk6iQbk1xaa/1gKeXdSV6T5PQkj06yoZSycBPitbXWFw2rJgAAWCtD3Ue61np5ksuXHHta/7fXxgfCAAAwoQRZAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABhuGefFSynOTvDrJvZK8vtZ68ZLzZyW5NMnmJH+T5MdqrUeGWRMAAKyFoc1Il1IelOTCJE9MclaS80opj1zytP+Z5MdrrQ9PMpXkR4ZVD2327juQ8y+5Ji987VU5/5JrsnffgXGXBADQCcNc2nFOkqtqrbfWWm9PckWSZy2cLKU8NMl9aq1/1z/0tiTPHmI9DGjvvgO5bPd1mT14KEkye/BQLtt9nTANAJDhBunTkty86PHNSR48wHnGbNee/Tl8ZO6oY4ePzGXXnv1jqggAoDuGuUZ6Osn8osdTSeYGOH9cMzMnNxd3T23ZcsrYxh6VW/sz0cc63sX338Wa0Jcu0pPu0ZNu0pfu6VpPhhmkP5vk7EWPtya5acn5B65w/uK23g8AAAXpSURBVLhmZ2/L3Nz88Z+4xrZsOSW33PKFkY87ag/YvOmuZR1Lj3ft/Z8oPZk0+tI9etI9etJN+tI94+rJ9PTUspO3w1zacWWSJ5dStpRS7pvk3CTvWThZa70hyb+XUp7QP/RDSXYPsR4GtHPHtmzccPQfkY0bprNzx7YxVQQA0B1DC9K11huTvCrJ1Uk+muTyWusHSynvLqU8tv+0H0jyulLKdUlOTvLGYdXD4LafuTXPe+ojMrN5U5JkZvOmPO+pj8j2M7eOuTIAgPEb6j7StdbLk1y+5NjTFv3+Y0keP8wauGe2n7lVcAYAOAafbAgAAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADTaMu4BGJyXJ9PTU2AoY59gcm550k750j550j550k750zzh6smjMk5aem5qfnx9tNWvjiUneP+4iAAA4YZyd5G8XH5jUIL0pyeOS3JzkzjHXAgDA+nVSkgcm+VCSQ4tPTGqQBgCAsXKzIQAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGG8ZdwHpQSvnKJG9IcnuSv6i1/smYS6KvlPLNSS6qtZ4z7lpOdKWUb0vyoiRTSfbUWi8dc0kkKaWcmeQXktyW5A9qrX815pJIUko5KcmVSc6vtV477npISiklyeVJPpHk2lrr68dc0gmvlHJGklcn+XyST9VaLxl1DWak18ZLk7y+1npekvPGXQw9pZSvSfL0JEfGXQtJklOT/GiS5yV55phr4ctOTvKyJK9M8pwx18KX/VySm8ZdBEd5YpLPJLkjyd4x10LPK5LckGRzkg+MowBBem1sTfLZcRfB0Wqtn6q1/nIE6U6otf5pksNJXpveT3DogFrr3ye5T5J3JnnPmMshSSnl3CT7kvzLuGvhKH+b3k/VXpLeT3EYv69LsivJTyW5YBwFCNJr4zNJHjjuIqDLSimnJrk0yTtqrVeOux56SimPSfJ/a61PSC8kMH7PSvKUJM9I8pNjroUve3ySjbXWwzFB0xUH0luW9sX0JmpGzhrptXFpkotKKV9K8jvjLgY66o1JHpzkZaWUz9Raf3bcBZEkuXeSS0spB5P8xbiLIam1PidJSikXJPnz8VbDItcl+c3+35XfHXcxJEku6v93a5K3jKOAqfn5+XGMOxFKKQtrbp5Ra/10/9hz01vYfq/01kVfPL4KT0z60i360U360j160k360j2T1BNLO5ZRSvmW9NZDPXzRsQcluTC9Gw7OSnJeKeWR46nwxKQv3aIf3aQv3aMn3aQv3TNpPRGkl/cj6d1QsPiu6XOSXFVrvbXWenuSK9Jby8bo6Eu36Ec36Uv36Ek36Uv3TFRPrJFeRq31RUnS2zbyLqcluXnR45vTu/mAEdGXbtGPbtKX7tGTbtKX7pm0npiRHsx0ksWLyqeSzI2pFr5MX7pFP7pJX7pHT7pJX7qnsz0RpAfz2Ry9zd3W2DC/C/SlW/Sjm/Sle/Skm/SlezrbE0s7BnNlkgtKKVvS+zjwc+OTDLtAX7pFP7pJX7pHT7pJX7qnsz0xIz2AWuuNSV6V5OokH01yea31g+OtCn3pFv3oJn3pHj3pJn3pni73xD7SAADQwIw0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIA61gp5YxSynwp5YeXHP/pUsrbxlQWwLogSAOsf3NJfqOUUsZdCMB6IkgDrH93JPmNJJeXUjaOuxiA9UKQBjgxXJjktiS/Mu5CANYLQRrgBFBrnUvyg0leUEr5znHXA7AeCNIAJ4ha62eS/GiSy5J85ZjLAZh4gjTACaTWekWS3UleNu5aACadIA1w4nlpkhvGXQTApJuan58fdw0AADBxzEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaPD/ARAod8u9E/cyAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot estimates\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "plt.figure(figsize=(12, 8))\n",
    "plt.scatter(n_values, z_estimates)\n",
    "plt.xscale('log')\n",
    "plt.xlabel('N')\n",
    "plt.ylabel('Z | Y = 1')\n",
    "plt.title('Empirical estimates of Z | Y = 1')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(c)**\n",
    "\n",
    "Fixing $Y := y \\in \\{0, 1\\}$, we get:\n",
    "\n",
    "$$ \n",
    "\\begin{align}\n",
    "\\mathbb{P}(Z = 1 | Y := y) &= p^*(z = 1) = \\sum_x p^*(x, z = 1) = \\sum_x p(x) p(z = 1 | x, y) \\\\\n",
    "&= \\frac{1}{2} \\frac{e^{2y-2}}{1 + e^{2y-2}} + \\frac{1}{2} \\frac{e^{2y}}{1 + e^{2y}}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "More explicitly,\n",
    "\n",
    "$$ \n",
    "\\begin{align}\n",
    "\\mathbb{P}(Z = 1 | Y := 0) &= \\frac{1}{2} \\alpha + \\frac{1}{2}\\frac{1}{2} \\\\\n",
    "&= \\frac{1}{4} + \\frac{\\alpha}{2} &\\approx 0.3096 \\\\\n",
    "\\mathbb{P}(Z = 0 | Y := 0) &= 1 - \\mathbb{P}(Z = 1 | Y := 0) \\\\\n",
    "&= \\frac{3}{4} - \\frac{\\alpha}{2} &\\approx 0.6904 \\\\\n",
    "\\mathbb{P}(Z = 1 | Y := 1) &= \\frac{1}{2} \\frac{1}{2} + \\frac{1}{2} \\left( 1 - \\alpha \\right) \\\\\n",
    "&= \\frac{3}{4} - \\frac{\\alpha}{2} &\\approx 0.6904 \\\\\n",
    "\\mathbb{P}(Z = 0 | Y := 1) &= 1 - \\mathbb{P}(Z = 1 | Y := 1) \\\\\n",
    "&= \\frac{1}{4} + \\frac{\\alpha}{2} &\\approx 0.3096\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(d)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def generate_samples_modified(n, y0):\n",
    "    seeds = np.random.uniform(low=0, high=1, size=(n, 3))\n",
    "    \n",
    "    x = np.where(seeds[:, 0] < 1/2, 1, 0)\n",
    "    y = y0 * np.ones(n, dtype=int)\n",
    "    z = np.where(seeds[:, 2] < np.exp(2*(x + y) - 2) / (1 + np.exp(2*(x + y) - 2)), 1, 0)\n",
    "    \n",
    "    result = np.zeros((n, 3), dtype=int)\n",
    "    result[:, 0] = x\n",
    "    result[:, 1] = y\n",
    "    result[:, 2] = z\n",
    "    \n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate data\n",
    "Xd = generate_samples_modified(n=10**6, y0=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Estimate with various segments of the data\n",
    "n_values = np.floor(np.logspace(0, 6, base=10)).astype(int)\n",
    "zd_estimates = [estimate_z_given_y(Xd[:k], 1, 1) for k in n_values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtIAAAH4CAYAAABntQpnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dfZxdd10n8E/SkAjbxsKYl6HAikb4VaJQoMDmBRjE6m4BBQOugmgBobA+sMiTICxbH4pofShIeQYJu9ZdFoIgEBYLJYBEoTz4EMhvIUCXQrPWKRhaISHM7B/3TrgZMpO5v94798zM+/169ZU559xzzvfON5N+5nd/55x1s7OzAQAAhrN+0gUAAMBKJEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA02DDpAgAWU0qZTfKPSb45b9Mjaq2fH8HxfyvJZ2qtbzjFtk8keVCt9SsNx31fkpfWWt80ghpfkOTvaq1vXazehuN+Z5K31FoffEuPtYRzvTzJf0hyZa31eQPrn5LkKfNfnuTPaq1PnHeMxyW5c631knnrr0hy7yT3r7V+s7/ujCTvT3J1rfX5I34vv5XkdrXWXxnlcYGVR5AGVoIfqbX+8zgOXGt9wSLbzhvHORs8OMknk8XrbXDbJPcd4fEW8+Qk/7bWet3gylrrK5K8Ym65lPLoJC9OcukQx35Gko8keW6S3+mve256v3z911tQ80lKKXdMcnmSC5P86aiOC6xcgjSwYpVSHpTkd5P83/RGMW9O8qIkT+0vv7nW+mv91/1ekmuTnJvka0keV2v9VCnl9Un+sdb6B6WUo0nemuQeSX4uvXC2pdb6z6WU5ya5KMnxJJ9O8rj+1y9PcpckU0m+muQxtda6SM3fmV5Q/KEkt0ryniTPqrUeL6X8ZpKfSnIsyXT/HLuSnJ/kslLKN5M8fKDeryf5oyQXJDkzySVJfrp/7C8l+Yla682llCekF2Q3JrldkhfVWl+eXhi8dX/k/d5J7tqvbSrJGUleUmt9XSnlzP5r75JkJslHkzy51joz771tT/LS/v6zSf6w1vqGUsoHkqxLsreU8ku11g8s8L25X5JX9uv+3ELfw/lqrV8vpTwmyV+XUt7eP9cvJbnP3Aj1vPM8Jcn5tdYnllKek+RnT3HYH621Ts9b94tJ3pfeLzW3W2p9wOpljjSwElxdSvnEwH9vGdh2n/SC4XlJjqQ3EvnQJPdK8sullHP6rzs/yZ/UWu+eXij8b6c4z8Ykf1lrLbXWa+ZWllJ+Mr1Qu6PW+oNJPpfkV9IbmfxKrXVHrfWu6QXv033c/8dJPlprvXeSeyb5riRPL6XcKcnT0gt/5yd5d5L71VqvSHJNemH7LfOOtSnJ4VrrfZPsTvKa/jHuluQ7kzy8H4KflOQhtdZ7JvmZJL/f3//xSb7W/96tS/KmJM/p17YzyTNLKf8uvXB/Vv919+nv+32DhZRSNiR5W771Pb4wyQtLKTtqrQ/sv+xHFgnRd0jyliTPqLXuO8338NvUWv8hyfP634PXJfnFWusXF3jtK+amjdRaX1RrPe8U/80P0am1/mat9aXp/TIBYEQaWBEWm9rxuVrrx/tfH0ryL7XWY0n+uZRyJN8aOfy7gRD3uiRXlFKmTnG8UwW9C5L8r1rrl5Ok1vr0uQ2llM+WUn41yfcneVCS/ad5Lw9Lct9Syi/2l2/d//MPkvxdko+VUvYm2Vtrfc9pjpUkb+7/eSjJP8yFx1LK59Kbx3tTKeVhSR5aSrlLkvPSG72e765JtiV5XSllbt2t0wv770ovFL8vyV8lubzW+plT7P8dtdY9SVJr/VIp5c3pzYte9HtSSrl1kr9I8sZa66uX8J5Pqdb6J6WURyX5ZK1171L2GXJEGuAkgjSw0h2dt/yNBV53fODrdf0/v+1j/yQ3LbDv7NxCKeXsJGenN+p6cXrTGa5McmOS7z1NvWck+ela66cGjjVba50ppexMb+T8giR/XEp5V6312ac53uD7/7b33p/Xuz/Jq5J8ML1R54ctUNe/DM4LL6V8d3/d10spc78oPDjJVaWUi2utfzlv/9mcbH1601dO57XpTWV5xhJeezqfS++XiiWptb4ovelAAEMztQNYK84rpdy9//XFST40xN04rkqyq5Syub98SZKnJ/n3SV5fa31tkprkJ9ILlIv530l+rZSyrpSyKb3pEL9SSrlHencn+VSt9XfTmwIyN43ieJYWSE/l/CQ3pHcR3rvTD9H9u1ocT3JGKWVdv/6vlVIe299+p3499y6l/Kf0psO8u9b66/33cK955zmY5BullF39/c9J8sj0RrAXVEr5jfRGyX/mVPOZAbrMiDSwElzdv9Bu0G8k+dchjnE4yaWllDsn+ackP7/UHWut7yyl3C29i9mS5EB6847vkeRV/Wka69Ib+f2h0xzuqeld0PcP6YXjq5L8fq31G6WUNya5ppRyU3oXRD61v8/bkvxuKWXjUmse8O4kT0gvKM8k2ZdesP7+JJ9J8uH++3lgehcyvriU8ux+bf+l1vrXc7cBTPLJUsq/pndx50sGT9Kv/xFJXlJKuSS9/7/8Vq316tPU99tJvphk38CUkiT5eK318Q3v97QGLzYcx/GBtWPd7Oz8T+IAVpf+XTte2r9QkBVqoftIA0yKqR0AANDAiDQAADQwIg0AAA0EaQAAaLBS79qxKb3bQl2fU98HFgAARuGMJLdP7+m1Jz27YKUG6fvk1E8fAwCAcXhgeg+2OmGlBunrk+TLX745MzPLf7Hk1NSZmZ4+1cPPmBQ96SZ96R496R496SZ96Z5J9WT9+nW57W3/TdLPn4NWapD+ZpLMzMxOJEjPnZtu0ZNu0pfu0ZPu0ZNu0pfumXBPvm06sYsNAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaLBhnAcvpWxO8qEkD6u1fn7etvOSvCbJ5iTvT/KUWuvxcdYDAACjMrYgXUq5X5JXJ7nrAi/570meWGv9m1LKa5M8KcnLx1XPKFz25x/Lp679yonlH/ies/OsR99r7Ofdf+Bw9uw7lOkjRzO1eVN27dyWHdu3LstxlrrPqGoEAFgpxjm140lJfjnJl+ZvKKV8T5Jb11r/pr/q9Ul+eoy13GLzQ3SSfOrar+SyP//YWM+7/8Dh7N57MNNHjiZJpo8cze69B7P/wOGxH2ep+4yqRgCAlWRsQbrW+sRa6wcW2HxOkusHlq9Pcsdx1TIK80P06daPyp59h3Ls+MxJ644dn8mefYfGfpyl7jOqGgEAVpKxzpFexPokswPL65LMLPDaBU1NnTmygm6JLVvOGtuxb+yP8p5q/TDnbTnOUvcZVY231HKei6XTl+7Rk+7Rk27Sl+7pWk8mFaSvS3L7geWtOcUUkNOZnr4pMzOzp3/hmN1ww1fHduzbbd50YsrE/PXDnLflOEvdZ1Q13hJbtpy1bOdi6fSle/Ske/Skm/SleybVk/Xr1y04eDuR29/VWq9N8vVSyv37q34+yd5J1LJUP/A9Zw+1flR27dyWjRtObtPGDeuza+e2sR9nqfuMqkYAgJVkWYN0KeWdpZTz+4s/l+SPSykHk5yZ5CXLWcuwnvXoe31baF6Ou3bs2L41F114bqY2b0qSTG3elIsuPHfoO2K0HGep+4yqRgCAlWTd7Ozkp0Y0uHOSz01qaoePe7pHT7pJX7pHT7pHT7pJX7qnA1M7vjfJ50/atuzVAADAKiBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAgw3jPHgp5TFJnp/kVkkur7VeMW/7hUl+r7/4D0meXGu9aZw1AQDAKIxtRLqUcocklyZ5QJLzklxcSrnbwPazk+xO8rO11rsn+bskLxxXPQAAMErjnNpxQZL31lpvrLXenORNSR41sP0uSa6ttX6yv/z2JI8YYz0AADAy4wzS5yS5fmD5+iR3HFj+dJI7lVLu0V/+j0m2jrEeAAAYmXHOkV6fZHZgeV2SmbmFWutXSim/kORVpZT1SV6d5NgwJ5iaOnMUdTbZsuWsiZ2bU9OTbtKX7tGT7tGTbtKX7ulaT8YZpK9L8sCB5a1JvjS3UEo5I8l1tdb79Zfvk+TQMCeYnr4pMzOzp3/hiG3ZclZuuOGry35eFqYn3aQv3aMn3aMn3aQv3TOpnqxfv27BwdtxBumrklxSStmS5OYkj0xy8cD22STvLqXcL72A/fQk/3OM9QAAwMiMbY50rfWLSZ6X5Ookn0hyZa31w6WUd5ZSzq+1ziR5cpJ3JalJvpzksnHVAwAAozTW+0jXWq9McuW8dQ8Z+PodSd4xzhoAAGAcPNkQAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAYbJl0Aa9v+A4ezZ9+hTB85mqnNm7Jr57bs2L510mUBAJyWIM3E7D9wOLv3Hsyx4zNJkukjR7N778EkEaYBgM4ztYOJ2bPv0IkQPefY8Zns2XdoQhUBACydIM3ETB85OtR6AIAuEaSZmKnNm4ZaDwDQJYI0E7Nr57Zs3HDyX8GNG9Zn185tE6oIAGDpXGzIxMxdUOiuHQDASiRIM1E7tm8VnAGAFcnUDgAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoMGGcR68lPKYJM9Pcqskl9dar5i3/V5JXplkY5IvJHlsrfUr46wJAABGYWwj0qWUOyS5NMkDkpyX5OJSyt3mvezFSV5Qa71HkprkmeOqBwAARmmcUzsuSPLeWuuNtdabk7wpyaPmveaMJJv7X98mydfGWA8AAIzMOKd2nJPk+oHl65Pcd95rnp7k3aWUy5PcnOR+Y6wHAABGZpxBen2S2YHldUlm5hZKKbdO8tokF9RaP1xKeXqSNyR56FJPMDV15ohKHd6WLWdN7Nycmp50k750j550j550k750T9d6Ms4gfV2SBw4sb03ypYHlH0zytVrrh/vLr0zy28OcYHr6pszMzJ7+hSO2ZctZueGGry77eVmYnnSTvnSPnnSPnnSTvnTPpHqyfv26BQdvxzlH+qokP1pK2VJKuU2SRyZ518D2zyS5Uyml9JcfnuQjY6wHAABGZmxButb6xSTPS3J1kk8kubI/heOdpZTza61fTvK4JG8spfx9kickefy46gEAgFFacGpHKeXj6d1V45RqrXc/3cFrrVcmuXLeuocMfL03yd4lVQoAAB2y2BzpX0/yP5L8QhKThAAAYMCCQbrW+u5SyhVJfqTW+oxlrAkAADrvdHOkL03y/uUoBAAAVpJFb39Xa/16krcuUy0AALBijPP2dwAAsGoJ0gAA0ECQBgCABqcN0qWUHxv8EwAAWNqI9O/N+xMAANa8YaZ2rBtbFQAAsMIsevs76JL9Bw5nz75DmT5yNFObN2XXzm3ZsX3rpMsCANYoQZoVYf+Bw9m992COHZ9JkkwfOZrdew8miTANAEzEUqZ2mNLBxO3Zd+hEiJ5z7PhM9uw7NKGKAIC1bilB+pnz/oRlN33k6FDrAQDG7bRButb6nsE/YRKmNm8aaj0AwLgt5T7SH1vKOhinXTu3ZeOGk/+6btywPrt2bptQRQDAWreUiw2ftMR1MDZzFxS6awcA0BWnDdK11o8uZR2M247tWwVnAKAzhnkgCwAA0CdIAwBAgwWDdCnF/aMBAGABi41Iv7eU8t3LVgkAAKwgiwXpjyT5eCnlx5arGAAAWCkWDNK11mcn+ZkkLy+lXFpKMZ8aAAD6Fr39Xa31A6WUeyT50/RGp68d2PaT4y4OAAC6aikPZPmFJD+S5GVJPjvecgAAYGVYMEiXUs5J8vokd0ry47XWjy9XUQAA0HWLzXv+xyTXJrm3EA0AACdbbGrHk2qtb162SgAAYAVZ7K4dQjQAACzALe0AAKCBIA0AAA0EaQAAaLDY7e+uTjK7yL7rkszWWh888qoAAKDjFrtrx+OWqwgYl/0HDmfPvkOZPnI0U5s3ZdfObdmxfeukywIAVoEFg3St9dqFtsFKsP/A4ezeezDHjs8kSaaPHM3uvQeTRJgGAG4xc6RZtfbsO3QiRM85dnwme/YdmlBFAMBqIkizak0fOTrUegCAYTQF6VLKulEXAqM2tXnTUOsBAIaxYJAupfyfUsrDFtj80THVAyOza+e2bNxw8l/xjRvWZ9fObROqCABYTRYbkT47yetKKc8+xTYj0nTeju1bc9GF554YgZ7avCkXXXiuCw0BgJFY7PZ3X0zy6CRvL6VsT/KkWuux/rbF7i8NnbFj+1bBGQAYi8VGpGdrrQeT3DfJHZO8r5Sypb/NiDQAAGvaaS82rLXemOTHk3w8yTWllPNiRBoAgDVusakdJ0ada63fTPLLpZS/T/KeCNIAAKxxi41IP3H+ilrrK5P8VJJPjK0iAABYARZ7RPgpb3FXa31/kgvGVhEAAKwAnmwIAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQIMN4zx4KeUxSZ6f5FZJLq+1XjGw7bwkrx94+ZYkX661/uA4awIAgFEYW5AupdwhyaVJ7p3kaJIPlVKurrV+MklqrZ9Icl7/tbdJ8uEkTxlXPQAAMErjnNpxQZL31lpvrLXenORNSR61wGufm2RfrfWDY6wHAABGZpxTO85Jcv3A8vVJ7jv/RaWU70xycZIfGmMtAAAwUuMM0uuTzA4sr0syc4rXPTbJX9Ra/2nYE0xNndlY2i23ZctZEzs3p6Yn3aQv3aMn3aMn3aQv3dO1nowzSF+X5IEDy1uTfOkUr3tEkhe2nGB6+qbMzMye/oUjtmXLWbnhhq8u+3lZmJ50k750j550j550k750z6R6sn79ugUHb8cZpK9KckkpZUuSm5M8Mr0pHCeUUtaldzHi/jHWAQAAIze2IF1r/WIp5XlJrk6yMclraq0fLqW8M8kLaq3XpHfLu2O11q+Pqw6AlWr/gcPZs+9Qpo8czdTmTdm1c1t2bN+64s4BsFqN9T7StdYrk1w5b91DBr7+p/SmfAAwYP+Bw9m992COHe9dWjJ95Gh27z2YJIsG3WGCces5AOgZa5AGoM2efYdOBNw5x47PZM++QyMLxi3nYO1ZDZ9atLyH9330C3n92w8seZ+Wcwy7TxfPsZw13XjkaG7Xsb+DgjRAB00fOTrU+mT4YNxyDtaWlk8tujYlqfU9vOFdNUe/8c0l7dN6jmH26eI5uljTchvnA1kAaDS1edNQ65Phg3HLOVhbFvvl7FTmQs/c37m50LP/wOFFz7P/wOE862V/nSe86L151sv+etHXD3uOYd/D3D5zIXop+7SeY5h9uniOLta03IxIwzyr4WNMVr5dO7edNAqTJBs3rM+undsW3Gdq86ZThuaFgnHLORI/IyvZsL0b9pezLk5JavnkZdh91uo5uljTchOkYUDXP0Ji7Zj7+zZM6Bk2GLecw8/IeI1zfmpL74b95ayLU5KGfQ8t+6zVc3SxpuVmagcM6PpHSKwtO7ZvzWW/dP+87jkPzmW/dP/TBtUd27fmogvPPfE/mKnNm3LRhecuut+w51gtPyPDTCVYrnMMO2VhOaY47Nq5LRs3nBwVFvvlrItTkoZ9D3P7bLrVGUvep/Ucw+zTxXN0sablZkQaBnT9IyQ4nR3bt451ZLjlZ2Q57gIwjK7eWnDYkdnlmOIw7KcWXZyS1PLJy47tW7P5rO9Y8l07Ws8xzD5dPMdy1+SuHdBxXf8IiZVtNcwtHvZnpItX6HdxHu/cMce5vvXft2F+OevqlKSWXzAfdO87Zfu/PXvJr285x7D7dPEcy1lTFx/bLkjDgNaLr+B0Vsvc4mF/RsY9ytqii/N4k/HPHV2uf99aQlIy/mAM4yBIw4CWf9AZvy6O5A5b02p5+MmwPyPLdYX+MP1oGZldjtHfYYPucozkLhfBmJVKkIZ5/IPeLV0cyW2paTXNvx/mZ2Q5rtAfth9dnMc7WOu456f69w1GR5AGOq2LI7ktNa3V+ffjHmVNhu9HV+fxzu037vmpwOgI0kCnLddI7jBTA1pqWqvz75djlLX1bhTm8QK3lCANLLtxz2dtqWeYqQEtNXV5fuq4jXuUdblG+wVjYD5BGlhWyzGfdVjDTg1orUkQG4+1OtoPTJ4gDSyr5ZjPOqxhpwas5dHlLtIPYFIEaWBZLcd81mG1TtUQ1LpDP4BJEKSBZbVc81mHmYdtagAALdZPugBgbdm1c1s2bjj5n55Rh9a5edhzgX1uHvb+A4dP+fod27fmogvPPRHmpzZvykUXnmuEE4BFGZEGltVyzGdtuc+zqQEADEuQBpbduEPranqKIADdZWoHsOos9vhpABgVQRpYdZZjHjYAmNoBrDruKwzAchCkgVXJxYMAjJupHQAA0MCINHCLDPPgEwBYTQRpoNncg0/m7tk89+CTJMI0AKueqR1As8UefAIAq50gDTTz4BMA1jJBGmjmwScArGWCNNDMg08AWMtcbAg08+ATANYyQRq4RTz4BIC1ytQOAABoIEgDAEADUzuAk3hSIQAsjSANnOBJhQCwdKZ2ACd4UiEALJ0gDZzgSYUAsHSCNHCCJxUCwNIJ0sAJnlQIAEvnYkPgBE8qBIClE6SBk3hSIQAsjakdAADQwIg0jIgHmQDA2iJIwwh4kAkArD2mdsAIeJAJAKw9gjSMgAeZAMDaI0jDCHiQCQCsPYI0jIAHmQDA2uNiQxgBDzIBgLVHkIYR8SATAFhbBGlYxdzbGgDGR5CGVcq9rQFgvFxsCKuUe1sDwHgJ0rBKubc1AIyXIA2rlHtbA8B4CdKwSrm3NQCMl4sNYZVyb2sAGC9BGlYx97YGgPExtQMAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCgwYZxHryU8pgkz09yqySX11qvmLe9JHllktsmOZzkZ2utXx5nTQAAMApjG5EupdwhyaVJHpDkvCQXl1LuNrB9XZK3JXlRrfUeST6e5DnjqgcAAEZpnFM7Lkjy3lrrjbXWm5O8KcmjBrbfK8nNtdZ39ZdfmOSKAADACjDOqR3nJLl+YPn6JPcdWP7+JIdLKa9Ncs8kn0ryq2OsBwAARmacQXp9ktmB5XVJZuad+0FJfrjWek0p5beT/FGSxy31BFNTZ97yKhtt2XLWxM7NqelJN+lL9+hJ9+hJN+lL93StJ+MM0tcleeDA8tYkXxpYPpzk07XWa/rLf57e9I8lm56+KTMzs6d/4Yht2XJWbrjhq8t+XhamJ92kL92jJ92jJ92kL90zqZ6sX79uwcHbcc6RvirJj5ZStpRSbpPkkUneNbD9Q0m2lFLu0V/+iSQfHWM9AAAwMmML0rXWLyZ5XpKrk3wiyZW11g+XUt5ZSjm/1vq1JD+V5NWllANJHpzkGeOqBwAARmms95GutV6Z5Mp56x4y8PXf5uQLEAEAYEXwZEMAAGggSAMAQANBGgAAGgjSAADQYKwXGwIL23/gcPbsO5TpI0cztXlTdu3clh3bt066LABgiQRpmID9Bw5n996DOXa897DP6SNHs3vvwSQRpgFghTC1AyZgz75DJ0L0nGPHZ7Jn36EJVQQADEuQhgmYPnJ0qPUAQPcI0jABU5s3DbUeAOgeQRomYNfObdm44eQfv40b1mfXzm0TqggAGJaLDWEC5i4odNcOAFi5BGmYkB3btwrOALCCmdoBAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANBGkAAGggSAMAQANBGgAAGgjSAADQQJAGAIAGgjQAADQQpAEAoMGGSRcALN3+A4ezZ9+hTB85mqnNm7Jr57bs2L510mUBwJokSMMKsf/A4ezeezDHjs8kSaaPHOLgesQAAAV7SURBVM3uvQeTRJgGgAkwtQNWiD37Dp0I0XOOHZ/Jnn2HJlQRAKxtgjSsENNHjg61HgAYL0EaVoipzZuGWg8AjJcgDSvErp3bsnHDyT+yGzesz66d2yZUEQCsbS42hBVi7oJCd+0AgG4QpGEF2bF9q+AMAB1hagcAADQQpAEAoIEgDQAADQRpAABoIEgDAEADQRoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAANNky6gEZnJMn69esmVsAkz82p6Uk36Uv36En36Ek36Uv3TKInA+c8Y/62dbOzs8tbzWg8IMkHJl0EAABrxgOTfHBwxUoN0puS3CfJ9Um+OeFaAABYvc5IcvskH0lydHDDSg3SAAAwUS42BACABoI0AAA0EKQBAKCBIA0AAA0EaQAAaCBIAwBAA0EaAAAaCNIAANBgw6QLWA1KKd+V5MVJbk7yjlrrWydcEn2llHsmuazWesGka1nrSik/nOSJSdYl2Vdrfc2ESyJJKWV7kt9MclOSP6u1/tWESyJJKeWMJFcleVat9ZpJ10NSSilJrkzyqSTX1Fovn3BJa14p5c5Jnp/kK0k+W2t92XLXYER6NJ6a5PJa68VJLp50MfSUUr4vyUOTHJ90LSRJzk7y5CQXJXn4hGvhW85M8rQkz0ny6AnXwrf8RpIvTboITvKAJF9I8rUk+ydcCz3PSHJtks1JPjSJAgTp0dia5LpJF8HJaq2frbX+TgTpTqi1vi3JsSQvSu8THDqg1vq3SW6d5M1J3jXhckhSSnlkkgNJPj3pWjjJB9P7VO2X0/sUh8m7S5I9SX4tySWTKECQHo0vJLn9pIuALiulnJ3kNUneWGu9atL10FNKuXeS/1drvX96IYHJe1SSH0/ysCT/ecK18C33TbKx1nosBmi64nB609L+Nb2BmmVnjvRovCbJZaWUbyR5xaSLgY56SZI7JnlaKeULtdbnTrogkiTfkeQ1pZQjSd4x6WJIaq2PTpJSyiVJ3j7ZahhwMMkf9X9WXjXpYkiSXNb/78Ykr51EAetmZ2cncd4VoZQyN+fmYbXWz/fXPSa9ie23Sm9e9BWTq3Bt0pdu0Y9u0pfu0ZNu0pfuWUk9MbVjAaWU+6U3H+quA+vukOTS9C44OC/JxaWUu02mwrVJX7pFP7pJX7pHT7pJX7pnpfVEkF7Yk9K7oGDwqukLkry31npjrfXmJG9Kby4by0dfukU/uklfukdPuklfumdF9cQc6QXUWp+YJL3bRp5wTpLrB5avT+/iA5aJvnSLfnSTvnSPnnSTvnTPSuuJEenhrE8yOKl8XZKZCdXCt+hLt+hHN+lL9+hJN+lL93S2J4L0cK7Lybe52xo3zO8CfekW/egmfekePekmfemezvbE1I7hXJXkklLKlvQeB/7IeJJhF+hLt+hHN+lL9+hJN+lL93S2J0akh1Br/WKS5yW5OsknklxZa/3wZKtCX7pFP7pJX7pHT7pJX7qnyz1xH2kAAGhgRBoAABoI0gAA0ECQBgCABoI0AAA0EKQBAKCBIA0AAA0EaYBVrJRy51LKbCnlF+etf2Yp5fUTKgtgVRCkAVa/mSR/WEopky4EYDURpAFWv68l+cMkV5ZSNk66GIDVQpAGWBsuTXJTkhdOuhCA1UKQBlgDaq0zSR6b5PGllB+bdD0Aq4EgDbBG1Fq/kOTJSXYn+a4JlwOw4gnSAGtIrfVNSfYmedqkawFY6QRpgLXnqUmunXQRACvdutnZ2UnXAAAAK44RaQAAaCBIAwBAA0EaAAAaCNIAANBAkAYAgAaCNAAANBCkAQCggSANAAAN/j8eyDXVS2w6ogAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot estimates\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "plt.figure(figsize=(12, 8))\n",
    "plt.scatter(n_values, zd_estimates)\n",
    "plt.xscale('log')\n",
    "plt.xlabel('N')\n",
    "plt.ylabel('Z | Y := 1')\n",
    "plt.title('Empirical estimates of Z | Y := 1')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 20.8.5**.  This is a continuous, Gaussian version of the last question.  Let $V = (X, Y, Z)$ have the following joint distribution:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "X &\\sim \\text{Normal}(0, 1) \\\\\n",
    "Y | X = x &\\sim \\text{Normal}(\\alpha x, 1) \\\\\n",
    "Z | X = x, Y = y &\\sim \\text{Normal}(\\beta y + \\gamma x, 1)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Here, $\\alpha$, $\\beta$ and $\\gamma$ are fixed parameters.  Economists refer to models like this as *structural equation models*.\n",
    "\n",
    "**(a)** Find an explicit expression for $f(z | y)$ and $\\mathbb{E}(Z | Y = y) = \\int z f(z | y) dz$.\n",
    "\n",
    "**(b)** Find an explicit expression for $f(z | Y := y)$ and then find $\\mathbb{E}(Z | Y := y) \\equiv \\int z f(z | Y := y) dy$.  Compare to (b).\n",
    "\n",
    "**(c)** Find the joint distribution of $(Y, Z)$.  Find the correlation $\\rho$ between $Y$ and $Z$.\n",
    "\n",
    "**(d)** Suppose that $X$ is not observed and we try to make causal conclusions from the marginal distribution of $(Y, Z)$.  (Think of $X$ as unobserved confounding variables.)  In particular, suppose we declare that $Y$ causes $Z$ if $\\rho \\neq 0$ and we declare that $Y$ does not cause $Z$ if $\\rho = 0$.  Show that this will lead to erroneous conclusions.\n",
    "\n",
    "**(e)** Suppose we conduct a randomized experiment in which $Y$ is randomly assigned.  To be concrete, suppose that \n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "X &\\sim \\text{Normal}(0, 1) \\\\\n",
    "Y | X = x &\\sim \\text{Normal}(\\alpha, 1) \\\\\n",
    "Z | X = x, Y = y &\\sim \\text{Normal}(\\beta y + \\gamma x, 1)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Show that the method in (d) now yields correct conclusions i.e. $\\rho = 0$ if and only if $f(z | Y := y)$ does not depend on $y$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**.\n",
    "\n",
    "**(a)**\n",
    "\n",
    "The joint distribution of $X$ and $Y$ is:\n",
    "\n",
    "$$ \n",
    "\\begin{align}\n",
    "f_{X, Y}(x, y) &= f_X(x) f_{Y | X}(y | x) \\\\\n",
    "&= \\left(\\frac{1}{\\sigma_X \\sqrt{2\\pi}} \\exp \\left\\{-\\frac{1}{2} \\left(\\frac{x - \\mu_X}{\\sigma_X}\\right)^2 \\right\\} \\right)\n",
    "\\left(\\frac{1}{\\sigma_{Y | X} \\sqrt{2\\pi}} \\exp \\left\\{-\\frac{1}{2} \\left(\\frac{y - \\mu_{Y | X}}{\\sigma_{Y | X}}\\right)^2 \\right\\}\\right) \\\\\n",
    "&= \\left(\\frac{1}{\\sqrt{2\\pi}}\\right)^2 \\exp \\left\\{ -\\frac{1}{2} \\left( x^2 + (y - \\alpha x)^2\\right) \\right\\} \\\\\n",
    "&= (2 \\pi)^{-2 / 2} \\text{det} (\\Sigma_{X, Y})^{-1/2} \\exp \\left\\{ -\\frac{1}{2} (v - \\mu_{X, Y})^T \\Sigma_{X, Y}^{-1} (v - \\mu_{X, Y})\\right\\}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "where\n",
    "\n",
    "$$\n",
    "v = \\begin{bmatrix}x \\\\ y\\end{bmatrix},\n",
    "\\quad \\mu_{X, Y} = \\begin{bmatrix}0 \\\\ 0\\end{bmatrix}\n",
    "\\quad \\text{and} \\quad\n",
    "\\Sigma_{X, Y} = \\begin{bmatrix}\n",
    "1 & \\alpha \\\\\n",
    "\\alpha & 1 + \\alpha^2\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "or, in words, the joint distribution of $X, Y$ is a multivariate normal with mean $\\mu_{X, Y} = 0$ and variance $\\Sigma_{X, Y}$.\n",
    "\n",
    "$$ (X, Y) \\sim \\text{Normal}( 0, \\Sigma_{X, Y}) $$\n",
    "\n",
    "This can be obtained by inspection, by defining the inverse variance matrix\n",
    "\n",
    "$$\n",
    "\\Sigma_{X, Y}^{-1} = \\begin{bmatrix}\n",
    "1 + \\alpha^2 & -\\alpha \\\\\n",
    "-\\alpha      & 1\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "from expanding $x^2 + (y - \\alpha x)^2$ and $(v - \\mu_{X, Y})^T \\Sigma_{X, Y}^{-1} (v - \\mu_{X, Y})$, taking mean $\\mu_{X, Y} = 0$ assuming the inverse matrix is symmetric, and equating the coefficients for the monomials $x^2, y^2, xy$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The joint distribution of $X, Y, Z$ then is:\n",
    "\n",
    "$$ \n",
    "\\begin{align}\n",
    "f_{X, Y, Z} (x, y, z) &= f_{X, Y}(x, y) f_{Z | X, Y}(z | x, y) \\\\\n",
    "&= \\left(\\frac{1}{\\sqrt{2\\pi}}\\right)^2 \\exp \\left\\{ -\\frac{1}{2} \\left( x^2 + (y - \\alpha x)^2\\right) \\right\\} \n",
    "\\left(\\frac{1}{\\sqrt{2\\pi}} \\right) \\exp \\left\\{ -\\frac{1}{2} (z - (\\beta y + \\gamma x))^2\\right\\} \\\\\n",
    "&= \\left(\\frac{1}{\\sqrt{2\\pi}}\\right)^3 \\exp \\left\\{ -\\frac{1}{2} \\left( x^2 + (y - \\alpha x)^2 + (z - (\\beta y + \\gamma x))^2\\ \\right) \\right\\} \\\\\n",
    "&= (2\\pi)^{-3/2} \\text{det}(\\Sigma_{X, Y, Z})^{-1/2} \\exp \\left\\{ -\\frac{1}{2} (v - \\mu_{X, Y, Z})^T \\Sigma_{X, Y, Z}^{-1} (v - \\mu_{X, Y, Z})\\right\\}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "where\n",
    "\n",
    "$$\n",
    "v = \\begin{bmatrix}x \\\\ y \\\\ z\\end{bmatrix},\n",
    "\\quad \\mu_{X, Y, Z} = \\begin{bmatrix}0 \\\\ 0 \\\\ 0\\end{bmatrix}\n",
    "\\quad \\text{and} \\quad\n",
    "\\Sigma_{X, Y, Z} = \\begin{bmatrix}\n",
    "1 & \\alpha & \\alpha \\beta + \\gamma\\\\\n",
    "\\alpha & 1 + \\alpha^2 & \\alpha(\\alpha \\beta + \\gamma) + \\beta \\\\\n",
    "\\alpha \\beta + \\gamma & \\alpha(\\alpha \\beta + \\gamma) + \\beta & -(\\alpha - \\beta\\gamma)^2 + (\\beta^2 + 1)(\\alpha^2 + \\gamma^2 + 1)\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "or, in words, the joint distribution of $X, Y, Z$ is a multivariate normal with mean $\\mu_{X, Y, Z} = 0$ and variance $\\Sigma_{X, Y, Z}$.\n",
    "\n",
    "$$ (X, Y, Z) \\sim \\text{Normal}(0, \\Sigma_{X, Y, Z}) $$\n",
    "\n",
    "This can be obtained by inspection, by defining the inverse variance matrix\n",
    "\n",
    "$$\n",
    "\\Sigma_{X, Y, Z}^{-1} = \\begin{bmatrix}\n",
    "1 + \\alpha^2 + \\gamma^2 & -\\alpha + \\beta \\gamma & -\\gamma \\\\\n",
    "-\\alpha + \\beta \\gamma  & 1 + \\beta^2            & -\\beta  \\\\\n",
    "-\\gamma                 & -\\beta                 & 1\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "from expanding $x^2 + (y - \\alpha x)^2 + (z - (\\beta y + \\gamma x))^2$ and $(v - \\mu_{X, Y, Z})^T \\Sigma_{X, Y, Z}^{-1} (v - \\mu_{X, Y, Z})$, taking mean $\\mu_{X, Y, Z} = 0$, assuming the inverse matrix is symmetric, and equating the coefficients for the monomials $x^2, y^2, z^2, xy, xz, yz$.  A symbolic mathematics library was then used to calculate the inverse and simplify it as above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have characterized the joint distribution of $X, Y, Z$, we can use theorem 15.5 to first compute the marginal distribution of $Y, Z$, and then use it again to compute the conditional distribution of $Z | Y = y$.\n",
    "\n",
    "The marginal distribution of $Y, Z$ is\n",
    "\n",
    "$$ (Y, Z) \\sim \\text{Normal}(0, \\Sigma_{Y, Z})$$\n",
    "\n",
    "where\n",
    "\n",
    "$$ \\Sigma_{Y, Z} = \\begin{bmatrix}\n",
    "1 + \\alpha^2 & \\alpha(\\alpha \\beta + \\gamma) + \\beta \\\\\n",
    "\\alpha(\\alpha \\beta + \\gamma) + \\beta & -(\\alpha - \\beta\\gamma)^2 + (\\beta^2 + 1)(\\alpha^2 + \\gamma^2 + 1)\n",
    "\\end{bmatrix} $$\n",
    "\n",
    "Now, the conditional distribution of $Z | Y = y$ is:\n",
    "\n",
    "$$ Z | Y = y \\sim \\text{Normal}(\\mu_{Z | Y} (y), \\Sigma_{Z | Y}(y)) $$\n",
    "\n",
    "where\n",
    "\n",
    "$$ \\mu_{Z | Y}(y) = \\mu_Z + \\Sigma_{zy} \\Sigma_{yy}^{-1} (y - \\mu_y) = \\frac{\\alpha(\\alpha \\beta + \\gamma) + \\beta}{\\alpha^2 + 1} y$$\n",
    "\n",
    "$$ \\Sigma_{Z | Y}(y) = \\Sigma_{zz} - \\Sigma_{zy} \\Sigma_{yy}^{-1} \\Sigma_{yz} = \\frac{\\alpha^2 + \\gamma^2 + 1}{\\alpha^2 + 1}$$\n",
    "\n",
    "The explicit PDF is\n",
    "\n",
    "$$ f_{Z | Y}(z | y) = \\frac{1}{\\sqrt{\\Sigma_{Z | Y}(y)} \\sqrt{2 \\pi}} \\exp \\left\\{ -\\frac{1}{2} \\frac{\\left( z - \\mu_{Z | Y}(y) \\right)^2}{\\Sigma_{Z | Y}(y)} \\right\\} $$\n",
    "\n",
    "Finally, the expectation is the mean,\n",
    "\n",
    "$$\\mathbb{E}(Z | Y = y) = \\mu_{Z | Y} (y) = \\frac{\\alpha(\\alpha \\beta + \\gamma) + \\beta}{\\alpha^2 + 1} y$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(b)**  The joint distribution of $X, Z$ once we set $Y := y$ is:\n",
    "\n",
    "$$ \n",
    "\\begin{align}\n",
    "f_{X, Z | Y := y}(x, z) &= f_X(x) f_{Z | X = x, Y := y}(z | x) \\\\\n",
    "&= \\left(\\frac{1}{\\sigma_X \\sqrt{2\\pi}} \\exp \\left\\{-\\frac{1}{2} \\left(\\frac{x - \\mu_X}{\\sigma_X}\\right)^2 \\right\\} \\right)\n",
    "\\left(\\frac{1}{\\sigma_{Z | X} \\sqrt{2\\pi}} \\exp \\left\\{-\\frac{1}{2} \\left(\\frac{z - \\mu_{Z | X}}{\\sigma_{Z | X}}\\right)^2 \\right\\}\\right) \\\\\n",
    "&= \\left( \\frac{1}{\\sqrt{2 \\pi}} \\right)^2 \\exp \\left\\{ -\\frac{1}{2} \\left( x^2 + (z - (\\beta y + \\gamma x))^2\\right) \\right\\} \\\\\n",
    "&= \\left( \\frac{1}{\\sqrt{2 \\pi}} \\right)^2 \\exp \\left\\{ -\\frac{1}{2} \\left( x^2 + ((z - \\beta y) - \\gamma x))^2\\right) \\right\\} \\\\\n",
    "&= (2 \\pi)^{-2 / 2} \\text{det} (\\Sigma_{X, Z})^{-1/2} \\exp \\left\\{ -\\frac{1}{2} (v - \\mu_{X, Z})^T \\Sigma_{X, Z}^{-1} (v - \\mu_{X, Z})\\right\\}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "where\n",
    "\n",
    "$$\n",
    "v = \\begin{bmatrix}x \\\\ z\\end{bmatrix},\n",
    "\\quad \\mu_{X, Z} = \\begin{bmatrix}0 \\\\ \\beta y\\end{bmatrix}\n",
    "\\quad \\text{and} \\quad\n",
    "\\Sigma_{X, Z} = \\begin{bmatrix}\n",
    "1  & \\gamma \\\\\n",
    "\\gamma & 1 + \\gamma^2\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "or, in words, the joint distribution of $X, Z$ is a multivariate normal with mean $\\mu_{X, Z}$ and variance $\\Sigma_{X, Z}$ (a result analogous to (a), but with a non-zero mean).\n",
    "\n",
    "$$ (X, Z) \\sim \\text{Normal}(\\mu_{X, Z}, \\Sigma_{X, Z})$$\n",
    "\n",
    "Therefore, from theorem 15.5, the marginal distribution of $Z$ is:\n",
    "\n",
    "$$ Z | Y := y \\sim \\text{Normal}\\left( \\beta y, 1 + \\gamma^2\\right) $$\n",
    "\n",
    "with a explicit PDF\n",
    "\n",
    "$$ f(z | Y := y) = \\frac{1}{\\sqrt{1 + \\gamma^2} \\sqrt{2 \\pi}} \\exp \\left\\{ -\\frac{1}{2} \\frac{\\left( z - \\beta y \\right)^2}{1 + \\gamma^2} \\right\\}$$\n",
    "\n",
    "and the expectation is the mean of the distribution,\n",
    "\n",
    "$$ \\mathbb{E}(Z | Y := y) = \\beta y $$\n",
    "\n",
    "Note that this is distinct from the result in (a) -- the relationship between $X$ and $Y$ is gone -- no $\\alpha$ is present in the definition of this distribution, since setting $Y$ \"breaks\" the arrow from $X$ into $Y$.  Accordingly, the distribution is the same as in (a) when setting $\\alpha = 0$, which also breaks that relation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(c)**  \n",
    "\n",
    "We already found the joint distribution between $Y$ and $Z$ during part (a); it is\n",
    "\n",
    "$$ (Y, Z) \\sim \\text{Normal}(0, \\Sigma_{Y, Z}) $$\n",
    "\n",
    "where\n",
    "\n",
    "$$ \\Sigma_{Y, Z} = \\begin{bmatrix}\n",
    "1 + \\alpha^2 & \\alpha(\\alpha \\beta + \\gamma) + \\beta \\\\\n",
    "\\alpha(\\alpha \\beta + \\gamma) + \\beta & -(\\alpha - \\beta\\gamma)^2 + (\\beta^2 + 1)(\\alpha^2 + \\gamma^2 + 1)\n",
    "\\end{bmatrix} $$\n",
    "\n",
    "From the variance matrix, we can extract the correlation $\\rho$,\n",
    "\n",
    "$$ \\rho = \\frac{\\text{Cov}(Y, Z)}{\\sigma_Y \\sigma_Z} = \\frac{\\alpha(\\alpha \\beta + \\gamma) + \\beta}{\\sqrt{(1 + \\alpha^2)(-(\\alpha - \\beta\\gamma)^2 + (\\beta^2 + 1)(\\alpha^2 + \\gamma^2 + 1))}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(d)**\n",
    "\n",
    "Note that we can have $\\beta = 0$ and $\\rho \\neq 0$.  In this situation, $Y$ and $Z$ are correlated but there is no causation between $Y$ and $Z$; we would still have $\\mathbb{E}(Z | Y := y) = \\beta y = 0$, so the conclusion would be erroneous."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(e)**\n",
    "\n",
    "In this new experiment, there is no relation between $X$ and $Y$ (given the empty set), as $Y$ is drawn from a distribution that does not depend on $X$.  $Z$ is then drawn from a distribution that depends on $X$ and $Y$.\n",
    "\n",
    "Let's determine the joint distribution of $X, Y, Z$ in this scenario:\n",
    "\n",
    "$$ \n",
    "\\begin{align}\n",
    "f_{X, Y, Z}(x, y, z) &= f_X(x) f_Y(y) f_{Z | X, Y}(z | x, y) \\\\\n",
    "&= \\left(\\frac{1}{\\sigma_X \\sqrt{2\\pi}} \\exp \\left\\{-\\frac{1}{2} \\left(\\frac{x - \\mu_X}{\\sigma_X}\\right)^2 \\right\\} \\right)\n",
    "\\left(\\frac{1}{\\sigma_Y \\sqrt{2\\pi}} \\exp \\left\\{-\\frac{1}{2} \\left(\\frac{y - \\mu_Y}{\\sigma_Y}\\right)^2 \\right\\} \\right)\n",
    "\\left(\\frac{1}{\\sigma_{Z | X, Y} \\sqrt{2\\pi}} \\exp \\left\\{-\\frac{1}{2} \\left(\\frac{z - \\mu_{Z | X, Y}}{\\sigma_{Z | X, Y}}\\right)^2 \\right\\} \\right) \\\\\n",
    "&= (2 \\pi)^{-3/2} \\exp \\left\\{ -\\frac{1}{2} \\left(x^2 + (y - \\alpha)^2 + (z - (\\beta y + \\gamma x))^2 \\right)\\right\\} \\\\\n",
    "&= (2 \\pi)^{-3/2} \\text{det} (\\Sigma_{X, Y, Z})^{-1/2} \\exp \\left\\{ -\\frac{1}{2} (v - \\mu_{X, Y, Z})^T \\Sigma_{X, Y, Z}^{-1} (v - \\mu_{X, Y, Z})\\right\\}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "where\n",
    "\n",
    "$$\n",
    "v = \\begin{bmatrix}x \\\\ y \\\\ z\\end{bmatrix},\n",
    "\\quad \\mu_{X, Y, Z} = \\begin{bmatrix}0 \\\\ \\alpha \\\\ \\alpha \\beta \\end{bmatrix}\n",
    "\\quad \\text{and} \\quad\n",
    "\\Sigma_{X, Y, Z} = \\begin{bmatrix}\n",
    "1 & 0 & \\gamma \\\\\n",
    "0 & 1 & \\beta \\\\\n",
    "\\gamma & \\beta & 1 + \\beta^2 + \\gamma^2\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "which can be constructed by:\n",
    "- getting the mean vector $\\mu_{X, Y, Z}$ from the expectations of the distributions ($\\mathbb{E}(X) = 0, \\mathbb{E}(Y) = \\alpha, \\mathbb{E}(Z) = \\mathbb{E}(\\mathbb{E}(Z | X, Y)) = \\alpha \\beta$)\n",
    "- constructing a symmetric inverse matrix $\\Sigma_{X, Y, Z}^{-1}$, expanding $(v - \\mu_{X, Y, Z})^T \\Sigma_{X, Y, Z}^{-1} (v - \\mu_{X, Y, Z})$, expanding $x^2 + (y - \\alpha)^2 + (z - (\\beta y + \\gamma x))^2$ and equating the monomial coefficients;\n",
    "- obtaining\n",
    "\n",
    "$$ \\Sigma_{X, Y, Z}^{-1} = \\begin{bmatrix}\n",
    "1 + \\gamma^2 & \\beta \\gamma & -\\gamma \\\\\n",
    "\\beta \\gamma & 1 + \\beta^2 & -\\beta \\\\\n",
    "-\\gamma & -\\beta & 1\n",
    "\\end{bmatrix} $$\n",
    "\n",
    "and inverting it using a symbolic mathematics library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then have,\n",
    "\n",
    "$$ (X, Y, Z) \\sim \\text{Normal}(\\mu_{X, Y, Z}, \\Sigma_{X, Y, Z}) $$\n",
    "\n",
    "The correlation between $Y$ and $Z$ becomes\n",
    "\n",
    "$$\\rho = \\frac{\\text{Cov}(Y, Z)}{\\sigma_Y \\sigma_Z} = \\frac{\\beta}{\\sqrt{1 + \\beta^2 + \\gamma^2}}$$\n",
    "\n",
    "so the correlation is 0 if and only if $\\beta = 0$.  \n",
    "\n",
    "On the other hand, if we set $Y := y$, the joint distribution becomes equivalent to the one in (b), where\n",
    "\n",
    "$$ Z | Y := y \\sim \\text{Normal}\\left( \\beta y, 1 + \\gamma^2\\right) $$\n",
    "\n",
    "This does not depend in $y$ if and only if $\\beta = 0$.\n",
    "\n",
    "Therefore, for this concrete example, the method in (b) yields correct conclusions if and only if $\\rho = 0$."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}