gpflow.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import gpflow\n",
    "import abc\n",
    "\n",
    "def catch_exception(func, *args, **kwargs):\n",
    "    try:\n",
    "        return func(*args, **kwargs)\n",
    "    except Exception as e:\n",
    "        return e"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <center><b>Towards GPflow 1.0</b></center>\n",
    "\n",
    "\n",
    "<center><p>Artem Artemev</center>\n",
    "<center><p>Cambridge, UK</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "### Agenda\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Use TensorFlow \"right\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* GPflow before."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* GPflow re-design."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Code examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* What's next?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <center><b>\"Simplicity is prerequisite for reliability.\"</b></center>\n",
    "\n",
    "_<center><p><p>-Edsger Dijkstra</center>_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <center><b>Use TensorFlow \"right\"</b></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### TensorFlow has two major terms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* It is only model definition... ?!\n",
    "* Model's symbolic representation via operations and tensors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Graph is **immutable**. (1)\n",
    "* Graph is a set of scopes - operations and variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* From (1) it follows that scopes are not modifiable too."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a = tf.get_variable('a', shape=())\n",
    "b = tf.get_variable('b', shape=())\n",
    "\n",
    "with tf.name_scope('c'):\n",
    "    c1 = a + b\n",
    "\n",
    "with tf.name_scope('c'):\n",
    "    c2 = a + b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "c1.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "c2.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Session\n",
    "\n",
    "* Works with only graph.\n",
    "* Executes graph operations.\n",
    "* Allocates resources, not the **graph**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### One model - one graph - many sessions !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Only **graph definition** matters in GPflow design."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* **Session** can be interchangeable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <center><b>GPflow before</b></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Status Quo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* One _flat_ TensorFlow variable contains all parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* TensorFlow parameter name scope is _virtual_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Parameter as a FSM (finite state machine). For e.g. parameter has `recompile` flag."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* **Next things deserve stand-alone slides!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Hacky functions\n",
    "\n",
    "* get_free_state\n",
    "\n",
    "* get_feed_dict_keys\n",
    "\n",
    "* update_feed_dict\n",
    "\n",
    "* make_tf_array\n",
    "\n",
    "* get_free_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Hacky functions\n",
    "\n",
    "* Most of the code exist just to make this possible:\n",
    "```python\n",
    "def obj(x):\n",
    "    self.num_fevals += 1\n",
    "    feed_dict = {self._free_vars: x}\n",
    "    self.update_feed_dict(self._feed_dict_keys, feed_dict)\n",
    "    f, g = self.session.run([self._minusF, self._minusG], feed_dict=feed_dict)\n",
    "    return f.astype(np.float64), g.astype(np.float64)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Not clear interface\n",
    "\n",
    "* When user should call `compile`?\n",
    "* What autoflow does with model internals?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <center><b>GPflow re-design</b></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Main goals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Use external graphs and sessions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* One param - one TensorFlow variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Optimization is not part of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Simple and clean design (as far as possible)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Glossary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* **Build** is a process of defining the _graph_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* **Compile** is a two phase process which consist of building and initialising for specific _session_. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Change"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* _Build_ status enum class: `YES`, `NO`, `NOT_COMPATIBLE_GRAPH`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* _ICompilable_ interface. It defines major properties and methods for compilable nodes in GPflow tree."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* _CompilableNode_ class joins two separate interfaces _ICompilable_ and _Parentable_. It implements common parts, provides useful methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* _AutoFlow_ class. It is not decorator anymore! The _autoflow_ decorator replaces it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* *param_as_tensor* decorator. _Explicit_ convertion from parameter to tensor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* _GPflowError_ exception."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Compilable interface\n",
    "\n",
    "```python\n",
    "class ICompilable:\n",
    "    __metaclass__ = abc.ABCMeta\n",
    "\n",
    "    @abc.abstractproperty\n",
    "    def graph(self):\n",
    "        pass\n",
    "\n",
    "    @abc.abstractmethod\n",
    "    def compile(self, session=None, keep_session=True):\n",
    "        pass\n",
    "\n",
    "    @abc.abstractmethod\n",
    "    def initialize(self, session=None):\n",
    "        pass\n",
    "\n",
    "    @abc.abstractmethod\n",
    "    def is_built(self, graph=None):\n",
    "        pass\n",
    "\n",
    "    @abc.abstractmethod\n",
    "    def _build(self):\n",
    "        pass\n",
    "\n",
    "    @abc.abstractmethod\n",
    "    def clear(self):\n",
    "        pass\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <center><b>Code examples</b></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Parameter. _Create - simple case._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a = gpflow.Param(10, name='a_param')\n",
    "\n",
    "print(\"Parameter graph: {graph}\".format(graph=a.graph))\n",
    "print(\"Parameter session: {session}\".format(session=a.session))\n",
    "print(\"Parameter variable tensor: {tensor}\".format(tensor=a.var_tensor))\n",
    "print(\"Parameter transformed tensor: {tensor}\".format(tensor=a.transformed_tensor))\n",
    "print(\"Parameter prior tensor: {tensor}\".format(tensor=a.prior_tensor))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Parameter. _Create - compilation_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a = gpflow.Param(10, name='a_param')\n",
    "a.compile()\n",
    "\n",
    "print(\"Parameter graph: {graph}\".format(graph=a.graph))\n",
    "print(\"Parameter session: {session}\".format(session=a.session))\n",
    "print(\"Parameter variable tensor: {tensor}\".format(tensor=a.var_tensor))\n",
    "print(\"Parameter transformed tensor: {tensor}\".format(tensor=a.transformed_tensor))\n",
    "print(\"Parameter prior tensor: {tensor}\".format(tensor=a.prior_tensor))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Parameter. _Create - other ways_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "tensor = tf.get_variable('b', shape=())\n",
    "b = gpflow.Param(tensor, name='b')\n",
    "b.compile()\n",
    "\n",
    "b.var_tensor is tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "tensor = tf.get_variable('c/variable', shape=())\n",
    "c = gpflow.Param(10, name='c')\n",
    "c.compile()\n",
    "\n",
    "c.var_tensor is tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Parameter. _Create - trainable flag_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "tensor = tf.get_variable('a/variable', shape=())\n",
    "a = gpflow.Param(tensor, name='a', trainable=False)\n",
    "\n",
    "catch_exception(lambda: a.compile())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Parameter. _Methods_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a = gpflow.Param(10, name='c_param')\n",
    "a.compile()\n",
    "\n",
    "a.read_value()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a.assign(11)  # NOTE: temporary assignement for specific session.\n",
    "a.read_value()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a.initialize()  # NOTE: drops previous assignment for particular session\n",
    "a.read_value()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Parameter. _Graph_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a = gpflow.Param(10, name='a')\n",
    "\n",
    "graph = tf.Graph()\n",
    "with graph.as_default():\n",
    "    a.compile()\n",
    "    \n",
    "print(\"Default graph: {0}\".format(tf.get_default_graph()))\n",
    "print(\"Param graph: {0}\".format(a.graph))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a.is_built(graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a.is_built(tf.get_default_graph())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Parameter. _Session_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a = gpflow.Param(10, name='a')\n",
    "graph = tf.Graph()\n",
    "session = tf.Session(graph=graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a.compile(session=session, keep_session=False)\n",
    "a.session is None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a.graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "with session.as_default():\n",
    "    a.compile()\n",
    "    \n",
    "a.session is session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Parameter. _Session - failures_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a = gpflow.Param(10, name='a')\n",
    "a.compile()  # Uses default graph\n",
    "\n",
    "session = tf.Session(graph=tf.Graph())\n",
    "catch_exception(lambda: a.compile(session=session, keep_session=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a = gpflow.Param(10, name='a')\n",
    "a.compile()\n",
    "\n",
    "session = tf.Session(graph=tf.Graph())\n",
    "catch_exception(lambda: a.set_session(session))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Parameter. _Multiple sessions_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "session1 = tf.Session()\n",
    "session2 = tf.Session()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a = gpflow.Param(10, name='a')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a.compile(session1)\n",
    "a.initialize(session2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a.assign(1e3, session=session2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(\"Parameter value for session1: {value}\".format(value=a.read_value()))\n",
    "print(\"Parameter value for session2: {value}\".format(value=a.read_value(session=session2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Parameterized objects. _Create_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "p = gpflow.params.Parameterized()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "tensor = tf.get_variable('tensor', shape=(), trainable=False)\n",
    "p.a = gpflow.Param(tensor, trainable=False)\n",
    "p.b = gpflow.Param(10)\n",
    "p.c = gpflow.DataHolder(10)\n",
    "print(\"p.a.full_name = {0}\".format(p.a.full_name))\n",
    "print(\"p.b.full_name = {0}\".format(p.b.full_name))\n",
    "print(\"p.c.full_name = {0}\".format(p.c.full_name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "p.compile()\n",
    "print(\"List of data holders: {0}\".format(list(p.data_holders)))\n",
    "print(\"List of trainable tensors: {0}\".format(list(p.trainable_tensors)))\n",
    "print(\"Prior tensor: {0}\".format(p.prior_tensor))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Parameterized objects. _Create - failures_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "tensor = tf.get_variable('tensor', shape=(), trainable=False)\n",
    "p = gpflow.params.Parameterized()\n",
    "p.a = gpflow.Param(tensor, trainable=False)\n",
    "p.b = gpflow.Param(10)\n",
    "p.c = gpflow.DataHolder(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "catch_exception(lambda: p.b.compile())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "p.compile()\n",
    "def assign_d(): p.d = gpflow.Param(20)\n",
    "catch_exception(lambda: assign_d())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Parameterized objects. _Change parameters_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(\"p.b before assign: {0}\".format(p.b.read_value()))\n",
    "p.b = 1e3\n",
    "print(\"p.b after assign: {0}\".format(p.b.read_value()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Decorators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class DumbModel(gpflow.model.Model):\n",
    "    def __init__(self, name=None):\n",
    "        gpflow.model.Model.__init__(self, name=name)\n",
    "        self.a = gpflow.Param(3.)\n",
    "        \n",
    "    @gpflow.autoflow()\n",
    "    def execute1(self):\n",
    "        return self.a.var_tensor * self.likelihood_tensor\n",
    "    \n",
    "    @gpflow.autoflow()\n",
    "    @gpflow.params_as_tensors\n",
    "    def execute2(self):\n",
    "        return self.a * self.likelihood_tensor\n",
    "\n",
    "    @gpflow.params_as_tensors\n",
    "    def _build_likelihood(self):\n",
    "        return -tf.square(self.a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "m = DumbModel(name='simple')\n",
    "m.compile()\n",
    "print(\"Execute 1 for {name}: {result}\".format(name=m.name, result=m.execute1()))\n",
    "print(\"Execute 2 for {name}: {result}\".format(name=m.name, result=m.execute2()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### GP regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "def plot_model_predic_f(m):\n",
    "    xx = np.linspace(-0.1, 1.1, 100)[:,None]\n",
    "    mean, var = m.predict_y(xx)\n",
    "    plt.figure(figsize=(12, 6))\n",
    "    plt.plot(X, Y, 'kx', mew=2)\n",
    "    plt.plot(xx, mean, 'b', lw=2)\n",
    "    plt.fill_between(xx[:,0], mean[:,0] - 2*np.sqrt(var[:,0]), mean[:,0] + 2*np.sqrt(var[:,0]), color='blue', alpha=0.2)\n",
    "    plt.xlim(-0.1, 1.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "N = 10\n",
    "X = np.random.rand(N, 1)\n",
    "Y = np.sin(12 * X) + 0.1 * np.cos(2.5 * X) + np.random.randn(N, 1) * 0.9 + 3\n",
    "\n",
    "k = gpflow.kernels.Matern52(1, lengthscales=0.2)\n",
    "m = gpflow.model.GPR(X, Y, kern=k)\n",
    "m.likelihood.variance = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "catch_exception(lambda: m.predict_f())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### GP regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "m.compile()\n",
    "plot_model_predic_f(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Cleaning\n",
    "\n",
    "* Just painful"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(\"GPR parameters: {tensors}\".format(tensors=list(m.trainable_tensors)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "lengthscales_tensor_name = m.kern.lengthscales.transformed_tensor.name\n",
    "\n",
    "m.clear()\n",
    "print(\"GPR parameters: {tensors}\".format(tensors=list(m.trainable_tensors)))\n",
    "print(\"GPR is built: {built}\".format(built=m.is_built_coherence()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Cleaning. _What if we re-compile it?_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(\"GPR is built: {built}\".format(built=m.is_built_coherence()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(\"Lengthscale tensor name before: {name}\".format(name=lengthscales_tensor_name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "m.compile()\n",
    "print(\"Lengthscale tensor name after: {name}\".format(name=m.kern.lengthscales.transformed_tensor.name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <center><b>What's next?</b></center>\n",
    "\n",
    "* HMC works.\n",
    "\n",
    "* GPflow optimizer interface.\n",
    "\n",
    "* Minibatch via queuing.\n",
    "\n",
    "* Compact project layout."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Optimizer Interface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class Optimizer:\n",
    "    def __init__(self, model):\n",
    "        self._model = model\n",
    "\n",
    "    @abc.abstractmethod\n",
    "    def minimize(self, *args, **kwargs):\n",
    "        raise NotImplementedError()\n",
    "\n",
    "class ScipyOptimizer(Optimizer):\n",
    "    def minimize(self, *args, **kwargs):\n",
    "        pass\n",
    "\n",
    "class TensorFlowOptimizer(Optimizer):\n",
    "    def minimize(self, *args, **kwargs):\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Compact project layout.\n",
    "\n",
    "\n",
    "```bash\n",
    "\n",
    "\n",
    "\n",
    "gpflow\n",
    "├── bases\n",
    "│   └── ...\n",
    "├── ekernels\n",
    "│   └── ...\n",
    "├── kernels\n",
    "│   └── ...\n",
    "├── models\n",
    "│   └── ...\n",
    "├── params\n",
    "│   └── ...\n",
    "├── training\n",
    "│   └── ...\n",
    "└── ...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <center><b>Questions ?!</b></center>\n",
    "\n",
    "![img](img/gpflow-logo.png)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}