diff_programming_boolean_func.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "diff_programming_boolean_func.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/luisitobarcito/DiffProgTF/blob/master/diff_programming_boolean_func.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "oeB8Dyg91NbB",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%matplotlib inline \n",
        "\n",
        "import tensorflow as tf \n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "mxSBaQsC1y59",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Learning  Boolean functions by example.\n",
        "Even though this might look like an overkill, it will be helpful to explain the concept of learning by example and the modular perspective. Let's say we want to learn two basic functions: the AND, and the OR functions."
      ]
    },
    {
      "metadata": {
        "id": "HsITzume1_Za",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Define a generic tuneable function.\n",
        "Our generic parameterized function module will model binary input to binary output vectors. Rather than producing binary digits, the generic function can output a real number between o and 1. These outputs can be interpreted as the probability of the output being 1, which also correspond to the expected value of the output under the Bernoulli distribution. Our function corresponds to an affine function followed by a squashing nonlinearlity such as logistic sigmoid.\n",
        "\\begin{equation}\n",
        "f(x) = \\sigma(Wx + b)\n",
        "\\end{equation}\n",
        "where $\\sigma(z) = 1/(1+e^{-z})$."
      ]
    },
    {
      "metadata": {
        "id": "zJHpqATI2Dsq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class GenFunc(object):\n",
        "    def __init__(self, n_in, n_out=1, act_func=None):\n",
        "        self.W = tf.Variable(tf.random_normal([n_in, n_out], stddev=0.35), dtype=tf.float32)\n",
        "        self.b = tf.Variable(tf.zeros([n_out], dtype=tf.float32), dtype=tf.float32)\n",
        "        if act_func is None:\n",
        "            self.act_func = tf.sigmoid\n",
        "        else:\n",
        "            self.act_func = act_func\n",
        "\n",
        "    def __call__(self, X, is_logit=False):\n",
        "        # For numerical stability, We can select logit output \n",
        "        # to train the function with cross-entropy \n",
        "        if is_logit is False:\n",
        "            return self.act_func(tf.add(tf.matmul(X, self.W), self.b))\n",
        "        else:\n",
        "            return tf.add(tf.matmul(X, self.W), self.b)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3Rnx-83u2Q5R",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Define a differentiable objective\n",
        "We use cross entropy function to evaluate the match of the output produced by our tuneable function and the desired output of our program. In the context of binary digits and our tuneable function representing the probability of being 1, the cross entropy function is also a likelihood function of the parameters given the desired input-output data.\n"
      ]
    },
    {
      "metadata": {
        "id": "oor9cV5g2SBA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def learnProgram(sess, f, train_in, train_out, n_iter=10000):\n",
        "    X = tf.placeholder(shape=train_in.shape, dtype=tf.float32)\n",
        "    Y = tf.placeholder(shape=train_out.shape, dtype=tf.float32)      \n",
        "    ## The cross entropy loss applied to the function to be tuned\n",
        "    loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=Y, logits=f(X, is_logit=True)))\n",
        "    ## Training procedure\n",
        "    trainer = tf.train.GradientDescentOptimizer(learning_rate=0.2)\n",
        "    train_step = trainer.minimize(loss)\n",
        "    tf.global_variables_initializer().run()\n",
        "    for iTr in range(n_iter):\n",
        "        sess.run(train_step, feed_dict={X: train_in, Y: train_out})"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "dxp_vQLa2b6W",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## The OR function\n",
        "The data for the OR function is given by the following truth table:\n",
        "\n",
        "| X1 | X2 | X1 OR X2 |\n",
        "|----|:--:|:--------:|\n",
        "| 0  | 0  |    0     |\n",
        "| 0  | 1  |    1     |\n",
        "| 1  | 0  |    1     |\n",
        "| 1  | 1  |    1     |\n"
      ]
    },
    {
      "metadata": {
        "id": "oeNlF8-F2gh5",
        "colab_type": "code",
        "outputId": "45e9bd9c-d5d0-4e89-e263-def03b65391e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        }
      },
      "cell_type": "code",
      "source": [
        "## Create our set of input output pairs\n",
        "X = np.asarray([[0, 0],[0, 1],[1, 0],[1, 1]], dtype=np.float32)\n",
        "Y_or = np.asarray([0, 1, 1, 1], dtype=np.float32)[:,None]\n",
        "## Let's display our set of input output pairs\n",
        "from tabulate import tabulate\n",
        "print(tabulate (np.concatenate((X,Y_or), axis=1), headers=['X1','X2', 'X1 OR X2']))"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "  X1    X2    X1 OR X2\n",
            "----  ----  ----------\n",
            "   0     0           0\n",
            "   0     1           1\n",
            "   1     0           1\n",
            "   1     1           1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "wC8pJqR428wf",
        "colab_type": "code",
        "outputId": "760325b7-ca4e-4c40-ab82-c82deb291c71",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        }
      },
      "cell_type": "code",
      "source": [
        "## Instantiate a tuneable function \n",
        "F = GenFunc(2, 1)\n",
        "## call the tuning routine  within a TF session\n",
        "with tf.Session() as sess:\n",
        "    learnProgram(sess, F, X, Y_or)\n",
        "    ## display the outputs of the function after tuning\n",
        "    Z = F(X).eval()\n",
        "    print(tabulate(np.concatenate((X,Z), axis=1), headers=['X1','X2', 'F(X)']))"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Colocations handled automatically by placer.\n",
            "  X1    X2       F(X)\n",
            "----  ----  ---------\n",
            "   0     0  0.0101954\n",
            "   0     1  0.995924\n",
            "   1     0  0.995925\n",
            "   1     1  1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "W-6HfVpC3Xt7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## The AND function\n",
        "The data for the AND function is given by the following truth table:\n",
        "\n",
        "| X1 | X2 | X1 AND X2 |\n",
        "|----|:--:|:---------:|\n",
        "| 0  | 0  |    0      |\n",
        "| 0  | 1  |    0      |\n",
        "| 1  | 0  |    0      |\n",
        "| 1  | 1  |    1      |"
      ]
    },
    {
      "metadata": {
        "id": "R-uDorH63bhN",
        "colab_type": "code",
        "outputId": "f6f843d1-b7c9-4d65-a2c6-44ef794df701",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        }
      },
      "cell_type": "code",
      "source": [
        "Y_and = np.asarray([0, 0, 0, 1], dtype=np.float32)[:,None]\n",
        "F = GenFunc(2, 1)\n",
        "with tf.Session() as sess:\n",
        "    learnProgram(sess, F, X, Y_and)\n",
        "    Z = F(X).eval()\n",
        "    print (tabulate(np.concatenate((X,Z), axis=1), headers=['X1','X2', 'F(X)']))"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "  X1    X2         F(X)\n",
            "----  ----  -----------\n",
            "   0     0  1.51419e-06\n",
            "   0     1  0.0101462\n",
            "   1     0  0.0101462\n",
            "   1     1  0.985793\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "EfARwIE13lAk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## The XOR function\n",
        "The data for the XOR function is given by the following truth table:\n",
        "\n",
        "| X1 | X2 | X1 XOR X2 |\n",
        "|----|:--:|:---------:|\n",
        "| 0  | 0  |    0      |\n",
        "| 0  | 1  |    1      |\n",
        "| 1  | 0  |    1      |\n",
        "| 1  | 1  |    0      |\n"
      ]
    },
    {
      "metadata": {
        "id": "LGqB6UTO3mUJ",
        "colab_type": "code",
        "outputId": "6669c625-e35e-4ae1-fc40-510ca2213f86",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        }
      },
      "cell_type": "code",
      "source": [
        "Y_xor = np.asarray([0, 1, 1, 0], dtype=np.float32)[:,None]\n",
        "F = GenFunc(2, 1)\n",
        "with tf.Session() as sess:\n",
        "    learnProgram(sess, F, X, Y_xor)\n",
        "    Z = F(X).eval()\n",
        "    print (tabulate(np.concatenate((X,Z), axis=1), headers=['X1','X2', 'F(X)']))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "  X1    X2    F(X)\n",
            "----  ----  ------\n",
            "   0     0     0.5\n",
            "   0     1     0.5\n",
            "   1     0     0.5\n",
            "   1     1     0.5\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "VhghvRUA3xaQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### What happened?\n",
        "Note that for the XOR the function tuning did not succed in replicating the input output relation.\n",
        "It turns out that our tunable function can only represent a subset of all binary functions. \n",
        "### Composing tunable modules to increase the expresivity\n",
        "Fortunately, our tunable function does represent the basic Boolean functions. Compositions of basic Boolean functions can give us all Boolean functions from 2 inputs to 1 output.  "
      ]
    },
    {
      "metadata": {
        "id": "Y0nvhNHN36qx",
        "colab_type": "code",
        "outputId": "fbb78c3f-8728-4788-8bfb-f60658225bf9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 573
        }
      },
      "cell_type": "code",
      "source": [
        "F1 = GenFunc(2, 1)\n",
        "F2 = GenFunc(2, 1)\n",
        "F3 = GenFunc(2, 1)\n",
        "## Define a composition of tunable functions\n",
        "def G(X, is_logit=False):\n",
        "    Z1 = F1(X)\n",
        "    Z2 = F2(X)\n",
        "    Z3 = F3(tf.concat((Z1, Z2), axis=1), is_logit=is_logit)\n",
        "    return Z3\n",
        "\n",
        "with tf.Session() as sess:\n",
        "    learnProgram(sess, G, X, Y_xor)\n",
        "    ## Show the tuned composition of functions G(X)\n",
        "    Z = G(X).eval()\n",
        "    print(\"\\n Tuned composition\")\n",
        "    print (tabulate(np.concatenate((X,Z), axis=1), headers=['X1','X2', 'G(X)']))\n",
        "    ## Show the tuned module F1(X)\n",
        "    Z1 = F1(X).eval()\n",
        "    print(\"\\n Tuned input module F1\")\n",
        "    print( tabulate(np.concatenate((X,Z1), axis=1), headers=['X1','X2', 'F1(X)']))\n",
        "    ## Show the tuned module F2(X)\n",
        "    Z2 = F2(X).eval()\n",
        "    print(\"\\n Tuned input module F2\")\n",
        "    print( tabulate(np.concatenate((X,Z2), axis=1), headers=['X1','X2', 'F2(X)']))\n",
        "    ## Show the intermediate tuned module F3(X)\n",
        "    print(\"\\n Tuned intermediate module F3\")\n",
        "    Z3 = F3(X).eval()\n",
        "    print( tabulate(np.concatenate((X,Z3), axis=1), headers=['Z1','Z2', 'F3(Z)']))\n",
        "   \n",
        "sess.close()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            " Tuned composition\n",
            "  X1    X2       G(X)\n",
            "----  ----  ---------\n",
            "   0     0  0.0284485\n",
            "   0     1  0.976936\n",
            "   1     0  0.97693\n",
            "   1     1  0.026026\n",
            "\n",
            " Tuned input module F1\n",
            "  X1    X2       F1(X)\n",
            "----  ----  ----------\n",
            "   0     0  0.00140859\n",
            "   0     1  0.0933116\n",
            "   1     0  0.0932741\n",
            "   1     1  0.882426\n",
            "\n",
            " Tuned input module F2\n",
            "  X1    X2      F2(X)\n",
            "----  ----  ---------\n",
            "   0     0  0.0611437\n",
            "   0     1  0.972814\n",
            "   1     0  0.972744\n",
            "   1     1  0.999949\n",
            "\n",
            " Tuned intermediate module F3\n",
            "  Z1    Z2       F3(Z)\n",
            "----  ----  ----------\n",
            "   0     0  0.0168781\n",
            "   0     1  0.992532\n",
            "   1     0  1.111e-06\n",
            "   1     1  0.00852749\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}