What_is_Entropy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What is *\"Entropy\"* ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Self-information "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before getting into *\"Entropy\"*, I have to wrap my head around *\"Self-information\"*.  \n",
    "Definition of *\"Self-information\"* $h(x)$ is  \n",
    "\n",
    "$$h(x) = -log(p(x))$$\n",
    "\n",
    "How come \"logarithm\" is used? As far as I know, there are two reasons.\n",
    "1. Decreasing function on the closed interval [0,1] is required.\n",
    "2. We want to do addition of *\"Self-information\"* . However, joint probability is calculated by product of probability. Hence　function which meet $h(p1 * p2) = h(p1) + h(p2)$ is required.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Entropy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "% matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*\"Entropy \"* can be said as expectation of Self-information.  \n",
    "To put it simply,*\"Entropy\"* describe unpredictability of event.  \n",
    "Let event be $X (X_{0},X_{1},X_{2},\\cdots,X_{n})$, and let probability of X be $P (P_{0},P_{1},P_{2},\\cdots,P_{n})$.  \n",
    "*\"Entropy\" $H(x)$* can be expressed as following.\n",
    "$$ H(x) = -\\sum_{i=1}^{n}P_{i}\\log_{2}P_{i}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def entropy(prob_list):\n",
    "    \"\"\"\n",
    "    parameter\n",
    "    ---------------------\n",
    "    prob_list : prob_list is expected to be numpy. This represents probability, hence \"prob_list.sum()\" should be 1.\n",
    "    \"\"\"\n",
    "    return (-prob_list * np.log2(prob_list)).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Example of *\"Bernoulli Trail\"*\n",
    "As an example, I'm gonna observe entropy in *\"Bernoulli trial\"*.\n",
    "Probability mass function of *\"Bernoulli trial\"* is described as bellow.\n",
    "\n",
    " $$ Ber(x|\\mu) = \\mu^{x}(1-\\mu)^{1-x}$$\n",
    "\n",
    "Now *\"Bernoulli Trial\"* can be depicted as following function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def bernoulli_trial(u):\n",
    "    np_prob = np.empty((2,))\n",
    "    np_prob[0] = (u ** 1)\n",
    "    np_prob[1] = (1-u)\n",
    "    \n",
    "    return np_prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate probability of *\"Bernoulli Distribution\"*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "prob_ber_list = np.arange(0.01,1,0.01)\n",
    "ber_result_list = np.empty((prob_ber_list.shape[0],2))\n",
    "\n",
    "# calculate probability of bernoulli\n",
    "for i, prob_ber in enumerate(prob_ber_list):\n",
    "    ber_result_list[i] = bernoulli_trial(prob_ber)\n",
    "    \n",
    "# calculate entropy\n",
    "entropy_list = np.empty((prob_ber_list.shape[0],),dtype=np.float64)\n",
    "for j, ber_result in enumerate(ber_result_list): \n",
    "    tmp = entropy(ber_result)\n",
    "    entropy_list[j] = tmp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Entropy of *\"Bernoulli Distribution\"* can be depicted as following graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x117fb9240>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAETCAYAAADUAmpRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VOXZ//HPlR2ysGQlBAir7JsBBEERUZGKLO5149G6\ndVHbqvXXp621Vp8+bW1r3epWtdYFASsu1A1BQNYAYd8hrCEEAklIyH79/phDnzESMkBmTmZyvV+v\neTFnzpkz3zMJ58q5zzn3LaqKMcaY5i3M7QDGGGPcZ8XAGGOMFQNjjDFWDIwxxmDFwBhjDFYMjDHG\nYMXABDER6SYiX4vIYhG5t868XBHJcitbQ0Rkqoh85DWtIpJ0Bus5o/f5uO5METnmj3WbpseKQTPg\n7BhzRGSh1+MOH97XVUSmBiDimfoF8L6qDgeedTtMYxGRp0Xkcn+8T0SuE5G+p5h/uYg8fbqffZL1\nJHsXaBEZJCLTz3a9xn8i3A5gAuZ7qpp9mu8ZBYwGXmv0NI2jE/ARgKrWuJyl0ajqj/z4vnuAX59i\nHf8G/n0mn19HH2AK8FdnvauAaxphvcZP7MigmXOOGn4gInNEZL2IvOi83g34OXC5cyQxzGnaeE1E\nnhCRac5y/UTkC2eZJSIyxWvd80TkAWf+ShGZKSJtRSRNRIpFJMVr2XUiMvYk+aY4610oIp+LSD/n\n9T8Cg4HHROSzejbvUhGZLyJrROQlEYl23hsuIn9w1rtMRJ4VkQhn3mvO9zFdRB4WkdEiskBE/uSs\na7OIXOGVr6Htv9pr+jURecCHn8lHJzsiE48nRGStiHwiIvfV9z4RmSgiy51cr4tIgvOX+iDgaRGZ\n5ZXxBhGZ7fx8rxaReXXW+6DTHLe+zl/7J90+EYkDngYGOZ8/UUSyRCTXa9kLnO91ofPvBV7zTvo7\nafxMVe0R4g8gF1gBzPN6RHnNm4bnD4MoYB8wxJk3FXjNaz1Tge3AZc50ArAbuMiZbudMn3j/PGAB\nEOdMP31ifcBLwP9zng8D1p0k91Bnfe2c6THALiDBa/1Xn2Kb3wEigXBgFvBrZ95DwDOAOI+XgAed\nea8By4B0Z3o0UAtMcKavANafxvZf7ZXpNeABr+/yI695CiQ5zz8Cpp5km24DFgPRzvQd9b0PWANk\nOc9HAWFemUZ7rXMeniOBE9/p1cA853mms/57nelE53u9yIftG31iPc50FpDrPO8EHAB6O9O9nelO\nDf1O2sN/DzsyaD7uUtXRXo9Kr3l/UdVa57VtQIdTrKdQVT91np8P5KnqXABVzQPeBq7yWv55VT1x\nEvJZYLzz/I/A90QkDM9O7q8n+ayrgLed9aKqX+LZaYzwcZufUtUq9TQhPe/12VOcdcx1HgOAnl7v\n+0pV93tNb1XVD53n6/m/78eX7W9MlwOvqmqFM/3KKZZ9BXjdOXpYr6q1p1j2I1UtrmdeOZ4ijqoe\nxlNgT/t8Rh3j8XzHG5z1bgC+qrPe0/mdNI3AzhkYgCqv58qpmw/z60yfrKdD8Xpee7LnqrpZRFYD\nk4BLgB/X83kNrf9UvD/bez3hwCNeO/i66m7jqb6fU+WruwOOqufzfFX3s+v9OanqUyIyA88RyCoR\nOV9V99azeN3trbMq9d5G7+dns30N/VxP53fSNAL7gs2pVAFx4GmvPsn8RUB7ERnpLJMK3AC877XM\n1BNt9cDdwGyveb8H/gZMV9Wyk6z/X8D1J84tiMiFQHvnc31xm4iEOUcfd3h99nvAQyIS76x3TN32\ndx81tP17gF7OvF54mpjOxgd4jqZaOtO/4Ns7ZJzPmwQcUtXHgdV4mmnA+ZnW8/M8mRYicqOzztbA\ndfzfCeZTbd+pfnc+AS4UkR7O/J54mpU+8TGT8QM7Mmg+XpZvXjM+V1V/2cB75gOPisgC4Ad1Z6pq\nkXMy9c8iEoWnff5BVV3stdga4D1nh74Lz075hOV4/ho86WWhqrpERB4EPhSRKqAST9t9UQO5wfPX\n5CI8O5hkPOcB/seZ9wc8O6pFIlIMHMRzlc1p8WH7HwfeFJFxwE7grC6tVNV/ikhXYKnzs3wbKKi7\nnFP8ugM/d763fcCJpr3peJp9vus8GpIDdBORz4FU4MkTzWKcevtWA5EishjP1UuHvbZjp4h8F3hV\nRBTP78B3VXWnD3mMn8g3jwCNaTzOVSnPqOqMeubfC4xU1WsDGswY8y12ZGBcISJ/xXMCdrLbWYwx\ndmRgjDEGO4FsjDEGKwbGGGMIonMGSUlJmpmZ6XYMY4wJKitWrDikqskNLRc0xSAzM5Ps7NPtZ80Y\nY5o3Ednly3LWTGSMMcaKgTHGGCsGxhhjsGJgjDEGKwbGGGOwYmCMMQY/FQNn6Lx3RWR3PfPvFc9w\ngzm+DANojDHGv/x1n0EB8H1gXd0ZInI+nj7fRzovfSki8/T0B2s3xu9UleLyag4Wl1NQUkHR8SpK\nyqspqaimorqG6hqlulYJE4gMDyMiTGgZHUFCTAQJMZG0iY0iNSGapLhoIsPtQNw0XX4pBqr6FUA9\n42dcgWfovkpnmb8DE4FvFQMRuRO4E6Bjx47+iGoMAOVVNWzJL2FTXglb8kvIPVzG7sJSdheWUV51\nqhEjfSMCqfExdExsSWZiSzonxdGzXTy92yWQEh9d3/8VYwLGjTuQE/EM6n1CHp4B0b9FVV8EXgTI\nysqy7lVNo1BVdh0uY1luIat2H2XV7iNsyS+h1vkNi44IIzMxlszEWC7onkxaqxhSEmJIjoumVYtI\nElpEEB8dSXSk50ggPExQharaWqpqlLKKaorLqykur6LwWCUHSyo4UFzO/qPH2XW4lLmbC3g3+/9G\noEyMjWJgh9YM7tSGQR1bM7hjG2Iiw136dkxz5UYxyAdSvKbTOPUYrMactaKyKuZtOciCrYdYvP0w\n+44eByAhJoJBHdtwae9Ueqcn0DMtgY5tWxIWdnp/qYtAdFg40REQFx1BSkIDeY5XsflACRvzilm7\nr4hVu48wZ9NBAKIiwsjq1IbzuyUxpmcKPdPi7cjB+J1fxzMQkQOqmlbntSzgT8DFeMZvnQM80NA5\ng6ysLLW+iczpyC8uZ/baPD5bn8+y3EJqapXWLSMZ3iWREd2SGN6lLV2S4k57x+8vRWVVZO8qZNH2\nw3y97RCbDpQA0L51C8b2SmF8v3YMyWzbZPKa4CAiK1Q1q6HlAnZk4AyBeL2qZovIB3jGpK0G3rGT\nx6axFJdX8dHqPD5YvY+lOwtRhR6pcdx1QRcu7pXKwA6tCW+iO9NWLSO5uFcqF/dKBeBgcTlfbjrI\nFxsPMi17D68v3kVaQgzj+7VjyuD29G3fyuXEJpQEzUhndmRg6qOqLN1ZyLvL9zB7XR7lVbV0SY7l\nygHpXNE/nW4pcW5HPGulFdXM2XSQD1fv56vNBVTW1NInPYFrszowaVB7WrWIdDuiaaJ8PTKwYmCC\nVlllNe+t3Mc/FueyJf8Y8dERXDkwnWuzOtA/o1XItrMXlVUxa/U+3s3ew7p9xbSIDGfy4PbcOjyT\nc9Li3Y5nmhgrBiZkFZRU8NqinbyxeBfF5dX0SU/g1hGZTOifTouo5nUVzrp9RfxjcS6zcvZTUV3L\nyG5J3DO6KyO6JoZsMTSnx4qBCTn7jh7n+XnbeDd7L1U1tYzrk8btIztzbqc2zX7Hd6S0kreX7+bV\nr3MpKKmgf0Yrvj+6G5f1SW32301zZ8XAhIwDReU8O3cb7yz39G5y1eAM7rygC12Sg/9cQGMrr6rh\nvZX7eHH+dnIPl9EnPYEfj+3Bxb1SrCg0U1YMTNArOl7Fc3O38eqiXGprlWuyOvDDMd1o37qF29Ga\nvOqaWmbl7OepOVvZXVjGwA6t+fn4Xgzt3NbtaCbArBiYoFVZXcs/l+zir19upeh4FZMHtuf+sT3o\nmNjS7WhBp6qmlpkr9vLnL7aQX1zBZX1S+dm4nnZU1YxYMTBBacHWAh75YD07CkoZ2S2J/ze+J33S\n7Xr6s1VWWc0rC3byt6+2U1lTy20jO3PvmO7ERrvRCYEJJCsGJqjsP3qc3368gdlrD9ApsSWPTOjN\nRedYO3djKyip4PefbGL6ir2kJcTw39/pxRX929n3HMKsGJigUFOr/HPJLn7/ySZqVPnhRd343qgu\n1lGbn63YdYRHPljHun3FXHROMr+d3M/OxYQoKwamyduaX8LPZq5h5e6jXNAjmccn9aVDWzsvECg1\ntco/Fufyh083A/DgZedw6/BM6/soxFgxME1WTa3y8oIdPPnZFmKjw3lkQh8mDky3pgqX7D1Sxn//\nax1fbSlgaOe2PHnNACvKIcSKgWmSdh8u46fTc1iee4RLe6fyxJR+JMVFux2r2VNVZqzYy6MfbkBV\n+eUVvbluSAcr0CGgyfVaasz7q/bxi/fXIcCT1wxgyuD2trNpIkSEa7I6MLxrIg9OX8PD761l3uYC\nfndVP1q3jHI7ngkAG5TV+F1pRTU/fXc190/LoWdaPP++fxRXnZthhaAJymjTkje/N4yfj+/JFxvz\nGf/UApbnFrodywSAFQPjV1vyS5jwzELeW7WXe8d04507zyOjjbVHN2VhYcKdF3Rl5j0jiAgP47oX\nFvPcvG0ES5OyOTNWDIzfzMrZx8Rnvqb4eDVvfm8YP7n0HCLC7VcuWAzo0JqP7x3J5f3a8ftPNnPn\nGysoOl7ldizjJ/Y/0zS6qppafv3Beu57J4e+7RP4+N6RjOia5HYscwbiYyJ55oZB/OqK3szddJAr\nn1nIpgPFbscyfmDFwDSqI6WV3PLKMl5blMvtIzvz1h3nkZoQ43YscxZEhNtGduadO8/jeGUNU55b\nxKfrD7gdyzQyKwam0WzJL+HKZxeyYvcR/nTtAH55RW8irVkoZGRltuXDH42ke0ocd72xgqfnbLXz\nCCHE/qeaRvHVlgKmPLeI8qpapt15HlMGZ7gdyfhBakIM0+4azqSB6Tz5+Rbun5ZDRXWN27FMI7D7\nDMxZe2fZbv77/XX0SI3n71OzaNfK+rgJZTGR4fz5uoF0T43nD59uJq+onBdvPtfuRwhydmRgzpiq\n8odPN/Hwe2sZ2S2J6XcPt0LQTIgIP7ioG09dP5Cc3Ue56vlF7CksczuWOQtWDMwZqa6p5YHpa3h2\n7nZuGNqBl2/NIs76xm92Jg5szxu3D+XQsUqmPL+IjXl2pVGwsmJgTlt5VQ13/3MFM1fu5f6x3Xli\ncj87UdyMDeuSyIy7hxMuwrUvLGbZTrtjORjZ/2BzWorLq7jllWXM2XSQ30zsw/1je1i3EobuqfHM\nuGc4yXHR3PzKUuZszHc7kjlNVgyMz46WVXLTy0tZufsIT10/iFuGZ7odyTQhGW1aMv3u4fRIjeeu\nN1bw8Zo8tyOZ02DFwPjk8LEKbnhpKZvySnjh5nO5ckC625FME5QYF81bdwxjYIfW/Ojtlfxr1V63\nIxkfWTEwDTpYUs71Ly5h56FjvHxrFhf3SnU7kmnC4mMief22oQzrnMhP3l3Nu8v3uB3J+MCKgTml\ngpIKvvvSUvYdPc6rU4dyQY9ktyOZIBAbHcGr/zWEUd2TeWjmGt7NtoLQ1FkxMPUqLPWcI9h35Div\nTh3C8K6JbkcyQSQmMpwXbz6XUd2T+NnMNdZk1MRZMTAndbSskhtfXkru4VJeuTWLYV2sEJjTFxMZ\nzku3ZDG8SyI/fXc1H6ze73YkUw8rBuZbjlVUc+vfl7G9wHOOYEQ3637anLmYyHBevjWLIZlt+fG0\nHD7fYJedNkVWDMw3lFfVcMfr2azbX8xz3x3MqO52jsCcvZZREfx96hD6tm/FD95ayaLth9yOZOqw\nYmD+o6qmlh++tYolOw/z5DUDGNvbrhoyjSc2OoLXpg4hM7Eld7yezeo9R92OZLxYMTCAp9O5h2eu\n5YuN+fzmyj5MGtTe7UgmBLWJjeKN24fRNi6KW19dxraDx9yOZBx+KwYicq2ILBORFSLyZJ154SLy\nlIgscZZ5XkQi/ZXFNOyPn21m5sq9/HhsD262O4uNH6UmxPDP24cRESZMfXUZB4vL3Y5k8FMxEJFO\nwGPAJUAWkCEiV3ktMh5or6rnqepQIBWY5I8spmFvLM51eh/tyL0Xd3M7jmkGOiXG8urUoRSWVjL1\n1eWUlFe5HanZ89eRwThgpqoWqWdcvBf45s5+LxAhImEiEgZUARvqrkRE7hSRbBHJLigo8FPU5u3T\n9Qf41QfrGdsrhccm9rFO50zA9MtoxXM3DmZzfgn3/HMlldW1bkdq1vxVDBIB7xGz84CUExOqugr4\nCvid85inquvrrkRVX1TVLFXNSk62q1oa25q9R7nvnVUMyGjN0zcMJsK6oTYBNvqcFH43pR8Ltx3i\nl++vszGVXeSv//35eO38gTTnNQBE5BYgSlUfUtWHgHgRuc1PWcxJ7D96nNtfzyYxNpqXbsmiRVS4\n25FMM3VNVgd+NKYb07L38OL8HW7Habb8VQxmA5NFJN6Zvg2Y5TW/D98cfzkK6O6nLKaO0opqbn89\nm+OVNfx96hCS46PdjmSauR+P7cF3+rfjd59s4tP1Bxp+g2l0fikGqpoHPAHMF5GlQL6qzhSReSKS\nBjwJDBWRRSKyBBgM/NEfWcw31dYq972zis0Hinnmu4M4Jy2+4TcZ42dhYcKT1wxgQEZr7n8nh3X7\nityO1OxIsLTRZWVlaXZ2ttsxgt4fP93MM3O38eiVfbh1RKbbcYz5hoKSCiY+sxARYdYPzycpzo5a\nz5aIrFDVrIaWszOGzcjHa/J4Zu42rh/SgVuGd3I7jjHfkhwfzQs3Z3HoWAXff3MlVTV2hVGgWDFo\nJjbsL+aB6asZ3LE1j9olpKYJ65fRit9f3Z9lOwv5zYffuuLc+ElEw4uYYHe0rJK7/plNQosI/nbT\nuURH2JVDpmmbOLA96/cX8+L8HfRr34prh3RwO1LIsyODEFdbq/x4Wg4Hisr5203nkpIQ43YkY3zy\ns3E9GdktiV/MWmcnlAPAikGIe3buNuZuLuBXE/owqGMbt+MY47PwMOGp6weSGBvFPW+uoKjMuqzw\nJysGIWzB1gL+9MUWJg9qz03DOrodx5jTlhgXzbM3DuZAUTk/eTeH2trguPoxGFkxCFF5Rce5750c\nuqfE8fjkvnbC2AStwR3b8MsrejNn00Ge/2q723FClhWDEFRdU8t9b+dQXlXD8zedS8sou07ABLeb\nz+vEhAHp/OnzLWTnFrodJyRZMQhBf52zlWW5hfx2Ul+6Jse5HceYsyYiPDG5LxltWnDv26s4Wlbp\ndqSQY8UgxCzadoin527jqsEZTBmc4XYcYxpNfEwkT98wiIJjFTwwfY31cNrIrBiEkMPHKrhvWg5d\nkmL5zcQ+bscxptH1z2jNw5f34ouN+by2KNftOCHFikGIUFV+NnMtRWVVPPPdwcRG23kCE5puOz+T\ni3um8D//3sTmAyVuxwkZVgxCxNvL9vDFxnweGncOvdoluB3HGL8REf736v4kxERw3zurqKiucTtS\nSLBiEAJ2FBzjsY82MLJbEred39ntOMb4XVJcNH+4egCbDpTwh082ux0nJFgxCHJVNbXcPy2H6Mgw\nnrx2AGFhdj+BaR4u6pnCzed14uWFO1m49ZDbcYKeFYMg9/SX21izt4j/mdyPVOt3yDQzPx/fi67J\nsTwwfTVFx627irNhxSCIrdtXxLNztzF5UHsu79fO7TjGBFyLqHD+dO1ACo5V8NhH1t312bBiEKQq\nqmv4ybs5JMVF8esJdhmpab4GdGjNPRd2ZcaKvczZmO92nKBlxSBI/eWLrWzJP8bvpvSnVctIt+MY\n46p7L+5Oz7R4Hn5vLUdK7e7kM2HFIAit2n2EF77azrVZGVzUM8XtOMa4LirCcwHFkdJKHvlgvdtx\ngpIVgyBTWV3Lz2auISU+hl9c0dvtOMY0GX3SW/GjMd35YPV+ay46A1YMgsxz87axJf8YT0zpS0KM\nNQ8Z4+2e0V05JzWeX7y/jpJyu7rodFgxCCJb8kt4du42rhyQzpieqW7HMabJiYoI43+v7k9+cTn/\n+8kmt+MEFSsGQaKmVvnZzDXERUfwyARrHjKmPgM7tOa/zu/MP5fsZtlOG/vAV1YMgsQ/FueyavdR\nHpnQh8S4aLfjGNOk/fTSHnRo24KHZ66hvMr6LvKFFYMgcKConCc/28IFPZKZODDd7TjGNHktoyJ4\nYnI/dhwq5YWvdrgdJyhYMQgCj320gcqaWh6b2MfGMjbGR6O6JzNhQDrPztvGzkOlbsdp8qwYNHFz\nNx/k47V5/OiibnRKjHU7jjFB5Zff6UV0eBi/mrXORkZrgBWDJqy8qoZfzVpHl+RY7rywi9txjAk6\nKQkxPDjuHBZsPcSHa/LcjtOkWTFowp75cht7Co/z20l9iY4IdzuOMUHpxmGd6J/Rit98uIFiu/eg\nXlYMmqjcQ6W8OH8HkwamM6JrkttxjAla4WHC45P6cbi0gqe+2Op2nCbLikET9dhHG4gMF/7f+F5u\nRzEm6PXLaMX1Qzrw+qJctubbuMknY8WgCZq76SBzNh3k3ou724A1xjSSBy/rSWx0BL/+cL2dTD4J\nKwZNTEV1DY9+uJ4uybH8l41nbEyjaRsbxU8v7cHX2w7zyboDbsdpcnwqBiJi/R8EyCsLd5J7uIxf\nT+hDVITVamMa03eHdqRnWjy//XgjxyvtzmRvvu5t/iIiX4rIdSIS4csbRORaEVkmIitE5MmTzO8n\nIp866/1IRDqcVvIQdLCknGe/3MYlvVO5oEey23GMCTkR4WE8emUf9h09zssL7M5kbz4VA1W9FLgZ\n6AR8LCKPi0in+pZ35j0GXAJkARkicpXX/HDgGeAmVR0D3A0cOeOtCBF/+mwLlTW1/NxOGhvjN8O6\nJHJ53zSe/2o7+cXlbsdpMk63HSLKeVQCL4jIL+pZbhwwU1WL1HOm5gVgktf8IUAe8ISILMRTDI7X\nXYmI3Cki2SKSXVBQcJpRg8uG/cVMy97DLcMz6Zxkdxob408PX96Tqppanvxss9tRmgxfzxl8ArwD\nbAHGquqjqjoOGFbPWxIB7zM0eYD3+IwdgeHAb4ALnOlb665EVV9U1SxVzUpODt1mE1Xl8dkbaNUi\nknvHdHc7jjEhr1NiLFNHZDJ9xV7W7y9yO06T4OuRwSOqOkpV31XV/5x1UdUJ9Syfzzd3/mnOaycc\nBb5S1T2qWgtMB849jdwh5ctNB/l622Huv7i7DW5vTID8cEx3WreI5LcfbbRLTfG9GNSKyCoR2SMi\nK0VkcAPLzwYmi0i8M30bMMtr/mKgv4icuLX2MiDH59QhpLqmlidmb6RLUiw3nlfvaRhjTCNr1SKS\nH1/Sg8U7DjNn40G347jO12LwLHCrqnYAbgKeO9XCqpoHPAHMF5GlQL6qzhSReSKSpqolwI+Bf4nI\nIiAaePWMtyKITV+xl+0FpTw0rieR4XYpqTGBdMPQjnROiuV/P9lETW3zPjrwde9TqqprAFR1A3C4\noTeo6puqOkhVh6nqA85ro1X1gPN8rtP0NEJV71DV6jPdiGB1vLKGP3++hcEdW3NZHxvT2JhAiwwP\n48HLzmHrwWPMXLnX7Tiu8rUYfCki94pIDxG5BdjgPO/hz3Ch7tVFOzlYUsHDl/eyQWuMccnlfdMY\n0KE1f/58S7MeItPXYjAGmIznEtH/wnPvwAvA3/yUK+QdKa3k+XnbGdsrhaGd27odx5hmS0R4eFxP\n8orK+cfiXLfjuManu4lV9SJ/B2lunpu3jdKKah68rKfbUYxp9oZ3TWT0Ock8O3c712V1bJZX9fl6\nn0GKiLwtIltE5F0RSfN3sFCWV3Sc1xfvYsrgDM5Ji2/4DcYYv3vosp4Ul1fxwvztbkdxha/NRC8B\nM4GBwFvAi35L1Aw88+U2VJX7LrYbzIxpKnqnJ3BF/3ReW5TLoWMVbscJOF+LQbyqzlDVMlV9H7D+\nEs7QnsIypi3fw/VDOtKhbUu34xhjvNw/tjvlVTX8bV7zOzrwtRiEiUgXAKd3URuQ9ww9NWcr4WHC\nD8d0czuKMaaOrslxTBmcwRtLdnGgqHl1YudrMfgpnt5KVwOfAw/4L1Lo2l5wjPdW7uXm8zrZCGbG\nNFH3Xdydmlrl2bnb3I4SUL4WgyJV7QWMUdWeqprtz1Ch6qkvthITGc7do7u6HcUYU48ObVty7ZAO\nvLN8N3uPlLkdJ2B8LQbPiIioaoN3HpuT25pfwodr9nPriEyS4qLdjmOMOYUfjemGiDSrowNfi8E8\n4H0R+aEzxsCdfswUkp6Zu40WkeHcMaqL21GMMQ1o16oF12V1YMaKvew7+q2hVkKSr8UgGliJZ5yC\ndni6pDY+2lFwjA9X7+fm4Z1oGxvldhxjjA9ONOc2lyuLfLoDGVisqp+dmBCRUX7KE5KenbudqIgw\nOyowJoi0b92Cq8/NYNryPfzgom6ktQrtiz58PTJ4+MQTEYkA/uCfOKFn9+Ey3s/Zx43DOtm5AmOC\nzPdHd6NGtVnclXzKYiAiI0VkJzBcRHY4zzcDawOSLgQ8N28b4WHCXRfYUYExwaZD25ZMHtSet5bu\n5mBJaN93cMpioKoLVbUz8GNV7aKqnVW1q6reEaB8QW3f0ePMWLGXG4Z0IMXuKzAmKP3gom5U1dTy\n8oKdbkfxK5+aiVT1byIyQEQuOPHwd7BQ8PKCHQDceaHdV2BMsOqcFMsV/dN5c8kuisqq3I7jN772\nWvov4EngDufxPX+GCgVHSit5Z9keJg5sT/vWLdyOY4w5C3df2JXSyhr+uXSX21H8xteriVJU9Xy/\nJgkxry/O5XhVDXdfaOcKjAl2vdMTuLBHMn9fuJPbR3YmJjL0umfz9WqiLSLS3q9JQkhZZTWvLcpl\nbK9UuqfaeAXGhIJ7RnflcGkl01eE5ljJvhaDLnjGPV4kIotFZJE/QwW7acv3cLSsinusDyJjQsaw\nzm0Z1LE1L87fTnVNrdtxGp2vxeAWoD9wA3C98685iaqaWl6av4OhmW05t1Mbt+MYYxqJiHD3hV3Z\nU3ic2esOuB2n0TV0n8EtAKq6C0/Ppbuc59cFIlww+nhNHvuLyrl7tJ0rMCbUXNIrla7Jsbzw1XZU\n1e04jarHQlRrAAASKElEQVShI4OpXs/f83o+rvGjBD9V5eWFO+iWEsfoHiluxzHGNLKwMOF7o7qw\nfn8xS3cWuh2nUTVUDMSH58axdGch6/YVc/vIzoSF2VdkTCiaPKg9bWOjQu4mtIaKgfrw3DheWbiT\ntrFRTB5kF14ZE6piIsO5aVhH5mzKZ+ehUrfjNJqGisFIEdkvInnA+V7PRwQgW1DZeaiULzbmc9Ow\njiF5DbIx5v/cNLwTkWFhvPp16BwdNNQ3UZSqpqtqO1WN9npuHe3U8erXO4kMC+Om4Z3cjmKM8bOU\n+BiuHJjO9Oy9HC2rdDtOo/D10lJzCkfLKpmevZcrB6aTEm910pjm4PaRnTleVcNby3a7HaVRWDFo\nBNOW7+F4VQ23nd/Z7SjGmADp1S6B87sl8o9Fu0LiJjQrBmepplZ5Y8kuhnVuS+/0BLfjGGMCaOqI\nzhwoLufzDfluRzlrVgzO0txNB9l75Di3jsh0O4oxJsDG9EyhfesWvL441+0oZ82KwVn6x5JdpCXE\ncEnvVLejGGMCLDxMuHl4J5bsKGRLfonbcc6KFYOzsKPgGPO3FHDjsI5EhttXaUxzdF1WB6IjwvjH\n4ly3o5wV24OdhTeW7CIyXLh+aEe3oxhjXNImNooJA9J5b+U+isuDdyQ0vxUDEblWRJaJyAoRefIU\ny70iIq/5K4e/lFZUMyN7L+P7tSM5PtrtOMYYF906PJOyyhpmBvFYB34pBiLSCXgMuATIAjJE5KqT\nLDcJiPJHBn97P2cfJRXV3DI80+0oxhiX9ctoxaCOrXlj8a6g7c3UX0cG44CZqlqknm/mBWCS9wIi\nkgo8ADxe30pE5E4RyRaR7IKCAj9FPX2qyptLdtOrXQKDO7Z2O44xpgm4+bxO7DhUypIdwdmbqb+K\nQSLgPfpDHlC3T+cX8BSD8vpWoqovqmqWqmYlJyc3fsoztHZfERvyivnu0A6IWO+kxhgY368dCTER\nvB2kdyT7qxjk882df5rzGgAichewQVWX+Onz/ertZbuJiQxjovVOaoxxxESGM2VwBp+sO0BhafD1\nV+SvYjAbmCwiJ0aDvw2Y5TX/MmCAiLwPvAiMEZE/+ilLozpWUc2snP1M6J9OQkyk23GMMU3IDUM7\nUllTy3srg+9Esl+KgarmAU8A80VkKZCvqjNFZJ6IpKnqFFX9jqpOAu4EvlTVB/yRpbF9kLOfssoa\nbhhml5MaY77pnLR4zu3UhreW7Q66E8l+u7RUVd9U1UGqOuzEjl5VR6vqgTrL5arqVH/laGxvL9tN\nz7R4BnWwE8fGmG+7YWhHdhSUsizIhsW0m85Ow9q9RazdV8QNQzvaiWNjzEl9p1874oPwRLIVg9Pw\nzvLdREeEMWmgnTg2xpxci6hwpgxqz+x1B4Jq4BsrBj4qr6rhg9X7ubxvGq1a2oljY0z9rsnqQGV1\nLR+uyXM7is+sGPjo8w35lJRXc/W5HdyOYoxp4vqkJ9AzLZ4ZQdQ9hRUDH81YsZf0VjEM75rodhRj\nTBMnIlx9bgar9xxla5B0bW3FwAcHispZsLWAKYMzCA+zE8fGmIZNGtSeiDBhRpDcc2DFwAf/WrWP\nWoWrzs1wO4oxJkgkxUUz+pwU/rVyX1CMkWzFoAGqyowVe8jq1IbOSbFuxzHGBJGrz83gYEkFC7Yd\ncjtKg6wYNCBnz1G2F5RyTZYdFRhjTs+Ynim0aRkZFCeSrRg0YMaKvcREhjG+Xzu3oxhjgkxURBgT\nB7bn8/X5FJU17VHQrBicQmV1LR+vzePS3mnEW6d0xpgzMGVweypravn3uqZ9z4EVg1NYsLWAo2VV\nTBqU7nYUY0yQ6te+FV2SYpmVs9/tKKdkxeAUZuXsp03LSEZ1bzoD6xhjgouIcOXAdJbsPMyBonrH\n8nKdFYN6lFZU8/mGfMb3a0dkuH1Nxpgzd+WAdFThozVN9+jA9nL1+GJjPseraphondIZY85Sl+Q4\n+me04v2cfW5HqZcVg3rMytlPeqsYsjq1cTuKMSYEXDkgnXX7itl28JjbUU7KisFJFJZWMn9LARMG\nphNm3U8YYxrBhAHpiMAHq5tmU5EVg5OYvTaP6lq1cQuMMY0mNSGGEV0T+SBnX5McEtOKwUl8sHo/\n3VPi6JkW73YUY0wImTigPbmHy1izt8jtKN9ixaCOg8XlLM8t5Ir+6Ta0pTGmUV3aJ5WIMGF2E7wB\nzYpBHZ+uP4AqjO+X5nYUY0yIad0yihHdkvhk3YEm11RkxaCO2WsP0C0lju6p1kRkjGl84/umsetw\nGRvyit2O8g1WDLwcPlbB0p2HGd/XjgqMMf5xaZ80wsOEf6894HaUb7Bi4OWzDfnUKlxuPZQaY/yk\nbWwU53Vpy+y1eU2qqciKgZfZa/PonBRrVxEZY/zq8r7t2HGolC35TecGNCsGjiOllSzafphxfdPs\nKiJjjF9d2icVEZpUt9ZWDByfb8ynplYZ39eaiIwx/pUSH8OQzLZN6ryBFQPHv9fmkdGmBX3bJ7gd\nxRjTDIzvm8bm/JIm01eRFQPgWEU1X287zLg+1kRkjAmMcU4rxGcbmsbRgRUDYMGWAiprahnbO9Xt\nKMaYZiKtVQx92ycwZ+NBt6MAVgwA+GLjQVq1iLTuqo0xAXVxz1RW7j7C4WMVbkexYlBTq8zdfJDR\n5yQTYSOaGWMCaGyvVFRh7uYCt6NYMcjZc4TC0kou7mVNRMaYwOrbPoHUhGjmbMx3O4oVgy82HiQi\nTLiwhw16b4wJLBFhTM9U5m8poKK6xtUszb4YzNmYz9DObWnVItLtKMaYZuiS3imUVtawZEehqzn8\nVgxE5FoRWSYiK0TkyZPM/5GILBGRxSLynIgEvDDtPlzGlvxj1kRkjHHNiK5JxESGud5U5JcdsIh0\nAh4DLgGygAwRucprfh9gAnC+qg4HkoEr/JHlVL5wvvyxvVIC/dHGGANATGQ4I7slM2fjQVc7rvPX\nX+PjgJmqWqSerXsBmHRipqquB65U1RONZBHA8borEZE7RSRbRLILChr/bPucTfl0S4mjU2Jso6/b\nGGN8NbZXCvuOHmdjXolrGfxVDBIB79vq8oBv/PmtquUi0lpE3gJyVPXzuitR1RdVNUtVs5KTG/cE\n77GKapbtLOTinnZUYIxx1xindWLuZvduQPNXMcjnmzv/NOe1/xCRvsA04ClVfdRPOeq1ZPthqmqU\nC8+xq4iMMe5KiY+hd7sE5m9x734DfxWD2cBkETkxMMBtwKwTM0UkGfgLcK2qLvVThlNasLWAFpHh\nnGt3HRtjmoBRPZJYufsIpRXVrny+X4qBquYBTwDzRWQpkK+qM0VknoikAdcBnYFZzmvzROROf2Sp\nz4JthxjWpS3REeGB/FhjjDmpUd2SqapRlu487MrnR/hrxar6JvBmnddGO0+fcR6u2HukjB0Fpdw4\nrJNbEYwx5huyMtsQHRHG/C2HGNMz8Je7N8ubzhZuPQTABd2TXE5ijDEeMZHhDOuSyIKt7pw3aJbF\nYMHWQ6QlxNAtJc7tKMYY8x8XdE9ie0Ep+49+60p7v2t2xaCmVlm47RCjuifZQDbGmCZlVHfP1Y0n\nWi8CqdkVg7X7iig6XsUo65jOGNPE9EiNIyU+mvkuNBU1u2KwYEsBIjCym50vMMY0LSLCqO7JLNx2\niJrawHZN0fyKwdZD9E1vRdvYKLejGGPMt1zQI4mjZVWs318U0M9tVsXgWEU1K3cfYaRdRWSMaaLO\nd1otFgT4vEGzKgYrdx2hulYZ0TXR7SjGGHNSSXHRnJMaz9KdgR3foFkVg+W5hYSHCYM7WhcUxpim\na0jnNqzcdSSg5w2aVTFYtrOQPukJxEb77cZrY4w5a0My23KsopqNecUB+8xmUwwqqmvI2XOUIZlt\n3Y5ijDGnNLSzZz+1LIBNRc2mGKzbV0RFda0VA2NMk9euVQsy2rRgea4Vg0a3bOcRAIZk2vkCY0zT\nNzSzLctzCwM2FGazKQbLcwvpmhxLYly021GMMaZBQzq35dCxSnYeKg3I5zWLYlBbq2TnFv6nHc4Y\nY5q6E03agWoqahbFYHN+CcXl1Xa+wBgTNLomx5IYG/WfJm5/axbF4ERltWJgjAkWIkJWZhs7MmhM\ny3YW0q5VDBltWrgdxRhjfDYksy27C8vILy73+2eFfDFQVZbnFjIks62NX2CMCSqBvN8g5IvBnsLj\n5BdXMMROHhtjgkzvdgm0jAoPSFNRyPfLUFlTw7g+aQzvYsXAGBNcIsLDuDarAx3btvT7Z0mgbmg4\nW1lZWZqdne12DGOMCSoiskJVsxpaLuSbiYwxxjTMioExxhgrBsYYY6wYGGOMwYqBMcYYrBgYY4zB\nioExxhisGBhjjCGIbjoTkQJg12m8JQk45Kc4TV1z3Xbb7ubFtts3nVQ1uaGFgqYYnC4RyfblrrtQ\n1Fy33ba7ebHtblzWTGSMMcaKgTHGmNAuBi+6HcBFzXXbbbubF9vuRhSy5wyMMcb4LpSPDIwxxvjI\nioExxpjgLwYicq2ILBORFSLy5Enm3+vMzxGRB9zI6A8+bPePRGSJiCwWkedEJOh/1ic0tO1ey70i\nIq8FMJpf+fAz7ycin4rIlyLykYh0cCNnYzvVdotIuIg85fyuLxOR50Uk0q2sjUlErhaRd0Vkdz3z\nG3ffpqpB+wA6AZuBVoAA04CrvOafDywGopzHQiDL7dwB2O4+wGdAuDM9HbjS7dyB2Hav5SYBbwCv\nuZ05QD/zcOArINmZzgDi3M4dgO2eAMzwmn4PuMbt3I207RfiucHswEnmNfq+Ldj/WhwHzFTVIvV8\nQy/g2QmccAXwqqpWqmol8Hdgogs5G9spt1tV1+PZ+dc4L0UAxwMf0y8a+pkjIqnAA8DjLuTzl4a2\newiQBzwhIguBuwmNn3lD270XiBCRMOfotwrY4ELORqeqX6lqfXcaN/q+LdiLQSJwwGs6D0g5jfnB\nqsHtUtVyEWktIm8BOar6eSAD+pEvP9MX8BSD8kCFCoCGtrsjMBz4DXCBM31rwNL5zym3W1VX4Tki\n+p3zmOf8MRTqGn3fFuzFIJ9vfgFpzmu+zg9WDW6XiPTFc0j9lKo+GsBs/nbKbReRu4ANqrok0MH8\nrKGf+VHgK1Xdo6q1eJoGzw1gPn9p6Od9CxClqg+p6kNAvIjcFuCMbmj0fVuwF4PZwGQRiXembwNm\nec2fBdwiIpEiEo7nL6UPApzRH0653SKSDPwFuFZVl7qQz58a+plfBgwQkffx3JwzRkT+GOCM/tDQ\ndi8G+otIkjN9GZATwHz+0tB298HTDHpCFNA9QNnc1Pj7NrdPkjTCSZYbgVXAUuCPzmvzgDTn+QPO\n/OXAT93OG4jtBn4IbHemTzzudDtzoH7mXstlEiInkH3ZbuAiYAGwCHgJiHA7s7+3G89fx7OcbV6C\n5wRyotuZG3n7D3g999u+ze5ANsYYE/TNRMYYYxqBFQNjjDFWDIwxxlgxMMYYgxUDY4wxWDEwzYSI\nVIrIPOcx17kpz9f3ZoqIzzexicivReTuk7x+tYj8xHn+moiMc55/JCJtneejfP0cYxpTRMOLGBMS\nClV1NICIXAo8AVwZyACqOqOe16/wmnwDz/0RxgSUHRmY5qgbUAwgIutF5C4RecWZfsir6+//9npP\ntIj8TUQWiMgMEYlxlv+V07XychEZ7bX8ABH5UERWn+geQUSmisjv6oYRkVwRiRGRR4E05+jlPBH5\n1GuZz0/naMaY02VHBqa5aCsi8/B09bwDz92bONPHVfV2ERkDjMXTPbACs5ymnE14/lqfqKq7nT71\nvy8ifwVygSygA/AsnjtEwdP18ESgJbBKRGY3FFBVHxGRW72OYOKcMQkUiFTVdWf1DRhzClYMTHPx\nn2aiOmKA953ng4FP1en6W0Q+wbOj3wRsVtUTg4zMAa7C8/9nAPA9oLbOeueop8O4YyKyijPrL+dp\nYCpQiacnVmP8xpqJjPHsbMHTsdtF4sBzlHCis7cuIpLoPL8QWI+nr/0uwGg8BUG81jkEQERaAv2B\nrT5m8R6lawZwsZNj5mlsjzGnzYqBMQ5V/QL42uuRo6ofObNXAH8Uka/wNBk9i6dTuGTgc+AWoNRr\ndQdF5EM8I1D9TlW9+54/lQ3OeYmeqlqNp9kpWz0DmBjjN9ZRnTFNmHMS+W5V3el2FhPa7MjAmCZK\nRD4CFlkhMIFgRwbGGGPsyMAYY4wVA2OMMVgxMMYYgxUDY4wxWDEwxhgD/H8+5rvRd5JvTwAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x117f63ba8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(prob_ber_list,entropy_list)\n",
    "plt.xlabel('Probability')\n",
    "plt.ylabel('Entropy')\n",
    "plt.title('Entropy of bernoulli distribution')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As your intuition, when probability mass function accords to uniform distribution, Entropy $H(x)$ become maximum:)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}