docs/source/example_notebooks/optimization_class.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimization Objectives."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using ``paragami``, a typical workflow is as follows:\n",
    "\n",
    "1. Define an objective function.\n",
    "2. Define parameter patterns for the input to the objective function.\n",
    "3. Flatten the input of the objective function. \n",
    "4. Use ``autograd`` to define derivatives of the flat objective.\n",
    "5. Optimize.\n",
    "\n",
    "To help reduce the boilerplate in step 4, ``paragami`` provides an ``OptimizationObjective`` class.  In addition to defining all the ``autograd`` derivatives, optimization objectives also provide basic logging and progress displays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import autograd\n",
    "import autograd.numpy as np\n",
    "import scipy as sp\n",
    "import example_utils\n",
    "import matplotlib.pyplot as plt\n",
    "import paragami\n",
    "import time\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem setup."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us consider again the multivariate normal objective.  The functionality in this section is described in previous notebooks.\n",
    "\n",
    "First, define some parameters and draw data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f030ffdeda0>]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJztnXtwXPWV57+/e/thh+Bx0XhWMcZ4EkgKb0RZjNCOhkERYxAWYNKgnZrZ8pY8RlhuY0NEJijxZtjxDrVSYh4RGDO0bYl111Izm1olNp61w8NjYRN1EDYyIwITAlNBIY4X0ymvYMB6dJ/9Q/r9uPf2vd23u6/UD51P1S2k7vv4dcuce+55fI8gIjAMwzCVg1bsBTAMwzDewoadYRimwmDDzjAMU2GwYWcYhqkw2LAzDMNUGGzYGYZhKgw27AzDMBUGG3aGYZgKgw07wzBMheErxkUvvvhiWrFiRTEuzTAMU7acPHnyQyJakm2/ohj2FStW4MSJE8W4NMMwTNkihHjPzX4cimEYhqkw2LAzDMNUGGzYGYZhKgw27AzDMBUGG3aGYZgKgw07wzBMhcGGnWEYxkA8Hkd3dzfi8Xixl5I3RaljZxiGKUXi8ThWr16NiYkJBAIBHDlyBPX19cVeVs6wx84wDDPDwMAAJiYmkEwmMTExgYGBgWIvKS/YsDMMw8zQ2NiIQCAAXdcRCATQ2NhY7CXlBYdiGIZhZqivr8eRI0cwMDCAxsbGsgzDAGzYGYZhTNTX15etQZdwKIZhGKbCYMPOMAxTYbBhZxiGqTDYsDMMw1QYbNgZhmEqDM8MuxBCF0IMCyH+0atzMgzDMLnjpcf+DQBveXg+hmEYJg88MexCiGUAbgGw14vzMQzDMPnjlcfeA6ATQMqj8zEMwzB5UrBhF0LcCuADIjqZZb92IcQJIcSJs2fPFnpZhmEYxgEvPPZrAdwmhPgVgH8A8KdCiP9p3YmIdhNRLRHVLlmyxIPLMgzDMHYUbNiJaBsRLSOiFQD+AsA/EdF/LnhlDMMwTF5wHTvDMEyF4am6IxENABjw8pwMwzBMbrDHzjDMvKISZppmg/XYGYaZN1TKTNNssMfOMMy8oVJmmmaDDTvDMPOGSplpmg0OxTAMM2+olJmm2WDDzjDMvCLfmabxeLxsbghs2BmGKTlKzYiWW9KVDTvDMCVFKRpRu6RrKXv9bNgZhikpvDKiXiKTrvJmk0/SdS5vWGzYGYYpKbwwol7jRdJ1Lm9YbNgZhikpSrVyJd+kq2Qub1iCiGbt5E7U1tbSiRMn5vy6DMOUHqWWKJ1NCv2sQoiTRFSbbT/22BmGKRr5xp3L9WZQqNfvFjbsDMMUjXzizqVYNVNqsKQAwzBFw02Lv1WNcb7ovRQCe+wMwxSNbIlSO++8FKtmSg027AzDFJVMcWc773zbtm0lWTVTSrBhZximZHHyzucqCVmuFGzYhRALABwDEJw53/8mor8p9LwMwzBe1bSXaxVNvnjhsY8D+FMi+lgI4QfwshDiMBH9zINzMwwzzynUO5+PVTQFV8XQNB/P/Oqf2ea+64lhmJIn33mjhcwpnY9VNJ7E2IUQOoCTAC4HsIuIXvHivAzDeEMphCLsPGcAWddVqMdtjdOHQiF0d3dXdFjGE8NOREkAq4QQiwH8WAjxVSJ6w7iPEKIdQDsALF++3IvLMgzjglIJRVg951gshn379mVdV6HiWcY4fSgUQkdHR9G/i9nG0wYlIjoH4CiANTbv7SaiWiKqXbJkiZeXZRgmA6USirA2IwFwtS4v5pTW19dj27ZtSCQSJfFdzDZeVMUsATBJROeEEAsB3Ajg+wWvjGEYTyiVhp76+nr09PSgv78fLS0tqK6uNnnsTusqtDLGGIYqle9itilY3VEIcRWAfQB0TD8B/JCI/jbTMazuyDBzS7nH2Asx6vlcs1SZM3VHIvpnADWFnodhmNmjFBp6nLpIM62r0PxAPtesBFgEjGGYvMi1BLGxsRE+nw9CCPh8PldhkELzA17E58sRlhRgGCZn8vWkZejXbQi40Jh4qU5jmm3YsDMMkzN2pYvZjOfAwACSySSICBMTE/jOd76DNWvWZDzGC8NcCmGouYZH4zEMkzNGj13XdRARpqamEAgEcPToUVtDGo/H8bWvfQ2Tk5PqNSEEFixYYOvxu02alkJieK7g0XgMw+RNNmNp9KSHhoawf/9+AMD4+DhisZjjMVdccQXefPNN9Zr03q1NR25DPaXSfFVqsGFnmHlOPB5HLBYDALS2tgKAK2MpQxybN292fZ1f/vKXaa/rup4WO3fbbVpoV2qlwoadYcocJ+/aTYgiHo/j+uuvx/j4OACgr68PN998M86fPw8iwqeffoodO3bgxz/+seN5W1tb0dfXh8nJSfj9fnVzsCJj7FZuvvnmtPW5SZrG43GMjo7C55s2Y/Op6iUrRDTn2x/+4R8SwzDTDA4OUldXFw0ODuZ17MKFC0nXdVq4cKE6h9PrVrq6ukgIQZhWZCUhBOm6rn6XWzQaJSKiaDRKPp+PNE1Lu162zxCNRtPOC4ACgYDtcZnOafx8gUCAIpFIXt9fuQHgBLmwseyxM0wRmY0GnPr6etchCukZS49d0zTbUsT+/n5UV1dj69atmJqaAjAdT5fndao8MXr3iUQCmqYhlUqZ9kkmk7brczsyD5gWFuQQzGewYWeYIiAN3ujoqG0DjtsqD6NhFkIgFAqZXnejw3L06FEVY6+pqUFHR4cKxUhaWlpsQynyek6f0XjT6unpQTAYxMTEBHw+H4gIyWQyrxDKfNF8yRc27Awzx1hLBY0x4lAolJMHL4W1tm7dimQyiY6ODlRXV6dJ1Ro7Nq03DTvPuL+/H0uWLMHZs2fR0tKC9vZ2xONxBINBjI+PI5VKgYhM17N+xu3bt6t9x8fH0d/fj56eHiQSCWWIjWvJpWxxvjYeucZNvMbrjWPszHymq6tLxbGFEBQOh1Us2fieruvU1dVlOtYu7pzpGGMsOhgMUiAQUHH3aDSadq5ssfnBwUFqamoiTdMyrnHhwoVqHxnDF0I4xvvd5gTmO+AYO8N4h5dNMI2NjdB1XXVhHj58GJ2dneq81hCDvLbTkAhjWMLn82F0dBTxeDwt1i5j20SE8fFxbNmyBURkOle2jtL6+nps374dx48fdwyDyHOkUikIIdTr8rp28XQuW/QYN9bf6409dqac8NqbHBwcpLq6OuXJ2nnZ0pM2XltWo2DG+41EIqZjIpGIySM3Hq9pGum6bnpSsLt+Jg/f6tm7qVjx+/2mqhufz8ceewHApcfOhp1hspAtPJIL1jCFtWzQ7tpGY24sRbSWCTqtU5YoGg2s3OyuH41GqampicLhcN6fWxr+aDSqPq/f71dlk5mOYaPujFvDzqEYhslCplBHrhjDFJqm4YYbbsD27dsBwHbAcigUMoVQrr32Whw/flxVlBhDFqFQSJUrGkMkw8PDKuxjRV5fniMej6twjzWxm0vliTEhW11d7SqMNR/FumYLNuwMkwVZgRGLxbB3715Eo1H09fW5jgNnGs0mjbpTJYyx9lvTNKxcuRKvvvpqWnxbGuRkMgkhBG666Sb1el9fn61RB6bLGIHPbirW+vCNGzdi+fLlBeUW2GDPPTxog2FcIA3T1NQUiEglFrMhSxsfeOABrF69GgBw5MgRPPjgg44JS2NpYmNjI4LBIHRdRzAYRGtra9rxgPlJIJlM4sCBA1i9ejVisZip9nzx4sWm9R0+fNi0vlAoZBpMUVPDw9HKES+GWV8KIAbg32E6brebiB4r9LwMUwnYGW3raLZMzTZO9dpO2iqysUjefM6cOaM8fiLCuXPnTMedPn3atL5EImGqf7/33nvVupzkeN0wn6R1SwI3gfhMG4AvALh65ucLAbwNYGWmYzh5ypQjg4ODFAwGSQhBwWDQVZLPbbWHF4lDWRkTDAZNVS2appGmaWnJ00AgoJKbxtr2SCRCkUiEwuGwaX9jFU6u6+KKF2/AXCVPiei3AH478/NHQoi3AFwC4M2MBzJMmSHb73P1PNevXw9gWhI3m56Km/eNvwPm7k2ptijlCvbs2aPi8z6fD6lUCj6fDxs2bDCtp7+/H6tWrcK9995r0o3xAq5RLwJurL/bDcAKAKMAFmXajz12Zj7gxlO17mPtBrV7302dud2xnZ2d1NTUZCo5tNacw6YcUgjhqMDo1ffAuANzXe4ohPg8gH4AHUQ0ZvN+O4B2YFqJjWEqGatWitFTNXrcRm92fHwcd999N1KplIppW73d/v5+205SO0/YqhcjyxhfeuklDA8PK89eno+IVEesRNd1tLW1OT5tuMEpT8Bx91nEjfXPtgHwA3gOwDfd7M8eO1PJZGpCyuSBW3XQw+GwKWaeq8duXI9R3wUG3Ra7GLuxK9ba4er1d8RefG5grjx2MS0G0QvgLSJ6tNDzMUwpkksc3KkJqb6+Ht3d3WlVKD09Pejv78f58+dx7Ngxdc6DBw8CmPaaN27cmBYTb2lpydr8Y52QJCGa1m0xVsHIc1RXV6tjiAh9fX0Feex2cNx9lnFj/TNtAP4E017APwM4NbPdnOkY9tiZciKXOHgwGKRwOOzoSRsnEAUCAQqHwyq2LatXrD87ablkitvL9UUiEdupRQDI7/c7esqRSMRRy2Y2vlP22N0B1ophGG8warBomkY+n89kkIzvYyZ84ff7qa6uzpSoNGq22AlkWY2u3c3BqB2jaZqjZK48zlqyqOu6+gzZdFtkOCnbvvnC2jC549aws6QAw1iwhl2MDUSapqnEpQwh2DUHTU5O4tVXX8XIyAiqq6sBwDRWLpVKqaYhO1KpFNauXYtPPvkELS0tJj0YmTRNpVJpE4ysIY6qqioEAgE1aHrnzp1q0EWmwdeZBnjMxnfMeAsbdoYx4DSD1K66RHaJyvd37NiB/fv3q3ORoVoFgKnaRNaVT05Ops0Ale8dOnQIyWQSx48fV0bVqh2TSCRMx1q7WFtbW1FTU6Ni8u3t7epzSn0YwF6rJpFIqJuYk466PFcuRrrQOa9MdtiwM4wBp6Se3OLxuG3DUX19PaqqqkznEkIoNciamho1Vk4IgbVr16K5uRmJRAKhUAjDw8MApmeOJhIJjI6OYvfu3WlGVWrHSPVFq9KkfEJYunQpOjs7AcBU5nj48GEAUDeNQCCA9evX22rVDA0NZXw6APIz0pw4nX3YsDOMgUy6LVYj1tramvFcV155Jd555x3s2bNHDXMeHh5GX18fnn32WRw8eBBr165FZ2en8qQl3/72t01GVWq8GJUme3t7EY1G0dvbi7a2NixatAg7duxQ52hubsbw8LAKESWTSdMTBQBMTEwAME9tknNXz58/r/azPh1kGsZtN//UKbTFg6hnBzbsDGMg05DkbJ5ma2sr+vr6VDy7oaEBv/jFL5BMJnH+/HkMDw9j+fLlmJqaUkZ7//79OHTokOlc3/72t/HQQw+Z1vXII49gbGxM3Uxee+01TE5OAgAmJyfx1FNPpUkA9Pb2Ynh42DGOL4RIU3A0Ni3J44QQCAaDJolgp2HcViOdLbTFMfbZgQ07M+9xGyOWgyzkLE8ZmojH40rCt6OjA6dOnVI15n19faqrs7e3F7fccotpDigwbZh37NiBTz75BEuWLMEzzzyTdu1kMql04IUQytM2Yo3VL1iwQCVrrfj9frS1taGmpsaUM2htbTV51Lqu48477zSFnYw3OCCzZnu20BYzS7gpnfF643JHplSwq0E3dnpaa9WN5YnBYJCi0SgFAgHHcXPGenAYSiGNXaDGn7Ntxlml2Tb5WeTvfr9fdbMayyftxt+5nWmarQad69W9BVzuyDD2OGm1WGPQRg/TGp4AgPHxcfT29qqQiMRYCtna2oqnn35adX7STKz7tttuw8GDB9O8bCd0XVf/JSJMTU0pXRfjmoxUVVXh6NGj6mnCrnvUKd6dyaPOJZTCYZfiwIadmVdYY749PT2mGnQjRkMnDeCnn35q2mfp0qXw+/2m0IgsVxwaGsJPfvITW8MvcTLKRjRNw1/91V9h8eLFCIVCuOeee9R71113HX7605/a1sTX1NRkDXnka3hzCaVw2GXu4dF4TEkg66rj8bgn+zlhjflKrZRNmzapJCAwbUzvuOMOdHR04Pbbb8fIyAjWr1+PhoYG0/mam5sxMDCAcDiMuro6rFu3DrW1tZicnMT+/ftx7NgxR6/c5/NBCJFV9zyVSqGnpweNjY2qtlx67ceOHYMQAtddd53pGCFEWo27E/X19WlTnZjyhj12pui4rYV2s1+2RKhd6EF6lGfOnFGhmFQqZUpi7t+/H5qmQdM0CCFARKYSwOeeew7j4+MYGhpy9ZmfffZZlUQVQqhzOuHU5QpMz2EdHR1NO8au7pyZH7BhZ4qO24YVu/3k642NjRgZGcGWLVuQSqUQDAYdbxDWBiNZ1fL2229nXKcMd/h8PiSTSWiahlAopNblNl4uzyWZmprCokWLMDaWNsbARCgUUq3+3/rWt/DRRx+p9/7t3/7NtC8ReS4DwJQPbNiZouO2YcW6n2ykkWV5MgEKwLYFXkrYGkv74vE4GhsbbcsH7SAiXH/99XjxxReVhkpPT09aCWOujI2NZfXaDx8+rBqcrHH7DRs24LHHHjPJ83JX5/yFDTtTdNwm8Kz7xWIxFZKw01ux3iBisZgyfOPj46p23GokZcxb0zR8+ctfBgC89dZbKvzy4osvquuNj49nbALKhQsuuAAff/yx4/uyisZ6LZ/Ph3A4jHA4jFgspmrnpZyBUXKAmR8IL/5B5kptbS2dOHFizq/LVA5WT1uWAxJNj3e77777sHjxYmXcBwYGMDQ0ZCpnlAbceFPw+/144oknlHaL7PSUTwaapmFqakoZV9noI3VdrCxbtgynT5/OKUyTD5FIBH/3d38HYPq72bFjBw4ePKhCR9Ymo0yw8mLpIoQ4SUS1WXd0U+zu9cYNSkyhGBtrpL651A7v7OxUWuJS91zXdaVvDoemHzm0ubOz09S0E41GVbORHKJh1Ck3apdbm4fcNhN97nOfy7pPpkamYDBoaqayDqaWo/CyNQhxQ1FpA25QYioZ2d5P9NkAZukV/+hHPzJVjci4OxHhC1/4Ak6fPm3SQdF1Xemjy/Z+STKZxKZNm9DQ0IBXXnkFU1NT8Pl8aG9vV978wMAAenp6kEgk8POf/9xUTUMz4ZtsHvsnn3xi+r2qqgpVVVU4deqUeu2qq67CG2+8oRK3l156Kd577z1V+mhspjJKBMt1uIm5WxPUsViMvfdyxI3193pjj50pBOt0H6OHDgcv2TicGTPebzAYpEgkogY6u23tl633g4ODFAwGlUxAOBymurq6tP1XrVrl2nOXW11dXdpYO7lWKUkgJzkZJQzk95NJ5sDNd5vLsGxm7oBLj50blJiCKbRpKNs54vE4Nm/ejNtvvx2bN29GLBZT5YVEhMWLF+PIkSO44YYblBcvG380TUMwGMQ111xjeu+LX/wiHn/8cbS2tqqB0rW12UOXklAopJKxRKQakuzq2M+dO2dKeAohEA6HsWrVKsfzf/TRR1i0aJHptUWLFqG/v189nUxNTSnvvaenx6QNf+edd6pKHTlQ241WukxQP/jgg9iwYYOqNDKWlzJlgBvrn20D0AfgAwBvuNmfPfbKwYuYbKZz2HmfPp8vo1CXceh0JBKx9cqFEGpgtPzZrVct49UNDQ2u9r/sssvSXuvs7KRwOOx4TU3T0maiOgmAWYdNy3yA3Xc0139bxlswxzH2/wHgCQAxj87HlAleTMPJdI6BgYG0csRkMom77rorTSrWWg4JwCRJ29PTg97eXgwNDRmdEgDuNFuM+3766ad4+eWXHfeR3rLf70dNTQ3ee+890/vGOL7sZk0mk6qWXT6NyBmrxjUKIbBkyRJ8+OGHAGAqawRgqu3fuHFj1moYpyoYFvAqXzwx7ER0TAixwotzMeWFF9NwMp2jsbExTWTL7/ebEpcSq4phd3e36YYxPDxsSkbmwqJFi7Bq1SoMDg6aBlID04b2oosuMmmzEBHC4bAaT3fw4MG0hKZE6rvL44znuPbaa3Hs2DHT/kSEDz74AACU4d+zZw/27dtnGnMHAMuXL89q1DPJNLCAV3kyZ1UxQoh2AO3A9D82pjLwwqvLdA5Z5RGLxXDmzBlUVVWZasvlDFHgs+qXvXv34q677kJNTY3phmHcxwkhBC677DL86le/Mr0+NjaGeDyOP/7jP8bx48dN3jMR2QpuPfvss2q9a9euTRtLZ8TpieH8+fPw+/1pTy0S6dkTEcbHx/Haa69lnGhkxYsnLqb08KxBacZj/0ci+mq2fblBqfjMRRNKIdeQx4ZCISQSibRzbN68GU899ZTj8XKQ9CWXXIKFCxeio6MDAHD33XdnNO5S79ypPNHoWQshcPHFF+PDDz/MGMrRdR0XXnihmlsKAA0NDfjwww/x5ptvqtcuuOCCNM0XKQH8R3/0R7byvH6/H0IINW5P7u+2ISmfYdRM8ZjzBiUAK8DJ07JgLpJihVxDlhHCobnGrgEn26ZpGgUCgZwSpLkkVHPZ/H6/SnBmur68tq7r1NTUpJqmNE1T5ZCDg4M0ODhITU1NKjFsTaa6+b6dpiUxpQW43JFxwkklsVjXsJY6fuc73zGJWdFMmMGo5mid5alpGlauXIlVq1YpeQEjcqoRWcoOrei6rsokr7rqKlxzzTVYt25dVs30XLjlllswMDCgQkXWz9HU1ISnnnoKCxYsgK7rCAQCaGlpga7rqozz6quvVh55fX09tm/fjmAwqPbPJdfBeuwViBvrn20D8PcAfgtgEsD7ANoy7c8ee3ExetOBQMA0A3M2ruHksUuv1dgE09nZaevB+nw+k8du9eit3rWdpy0lBmRj0rp169LKBtetW5d2rJQZ8MJ7lxIHsqmpoaGBrrzySiV7INcmPXHpSVs/s13DEXvelQ9ceuyehWJy2diwFx+vap2zXSPbQGSjsdR1nZYtW2Yblujs7KSuri6KRqPqv1K7xW3H6IoVK5TRjEajdNFFF5neX7JkiaOWTFdXl21XabZtyZIlpvVdfvnljjcuqUNj9/cwauMYv69cQi5M+ePWsLNWzDxFVpvI7sXZqIjIVCq3Y8eOtPmhAHD69Om014QQ6OnpMSUI/X4/NmzYAACulRN/9atfoa+vD4sWLTLVkUvOnj1re5yu6wiFQli6dKmr62Q65zvvvGO7XzKZxOnTpzE5OQmidF0XWRI6Pj6uvoN8y0uZyodle+cxxaqI2L17NzZt2mR6LduQCTvsar8lUj7A7r1LLrkEv/nNb3K6jpyaZKxdF0J4Jsfr9/tBRCp3EAwGcfTo0bRBIZkqhZjKx21VDHvs8xhj/bgc8SZft8OrEsn+/v6016wGWJYdOhlnu2MkwWAQjz/+OPr7+/HCCy+k7ef3+02/Z7upEFFaHXmmdVmRnaV2QzJWrlypBmTv2bNHvd7c3Oz4HfO4OyYrbuI1Xm8cYy8trGqJ0Wg0bZ9oNEo+ny+jSqA12ecUX49Goxnj0jKxmE+ysqGhIU07JlMMXsa2w+Gwa+0Xp23FihW255ea7V1dXSatd6uGulETx/ie8bOwbsv8Bpw8ZdzS1dVlMn6yzlpirRvXNC0taWeVe5WG2Wq8pLGPRqO0ePFiR8OebwWKNHoyOdzQ0EB1dXW0bt06WwNvlL+VErz5XNfv9ytJXfmaHNJhxemmFw6HHROjxuQpJ03nL2zYGdcMDg6Sz+dzNNxWw28sPZTHGxtkrMZRVqJYPU6nBh1ZlpiPgdV1XZVQWs/ptL/bqhrrzaeurs5kbGWVjlH7PRKJuP4bsMfOZIMNO5MTxgEOdvXRMqQhPVO795yMckNDg21nZDQadTSqhXjskUjE8Xjr63LgRq7G3efzpYVu5BOK8Sbp5LUbsbsx2t0QuE6dYcPO5Ey2unOpbe5UX61pGjU1Nal4vJ1Bla39smY7HwNubSyS22WXXabCPE4ev91sUafJR06b1HG3u4au61RXV5dWn+9kjO1ujOyRM064NewsKcAosrWW79u3D3v27MHq1atV+7+sr9Z1HcFgEC0tLUgkErj11ltNLftE09ritbW1EELgwIEDaS3+btA0LU0oS/L+++9jZGQEHR0dSgp32bJlpnVYZ4sCwKlTpzA8POzq+la9dKN8gawtb2trM0kQJJNJJSlsRUovyNp0t5OOGCYTXO7IuMJJ3tVaMtnR0aGkdI2133JE3dKlS/Hqq6/mbNAlqVQKb7/9tu17yWQS999/v6nx6f333896TqNEb6baeCGESU1RDu9IJBJpteXPPPNMmo66HVYt+u3bt7NRZwqGDfs8opA69MbGRui6jlQqBV3XTR2P0sB3d3erzkhJe3s7ampqkEgkcO7cOTzyyCPKaEpv166+OxNf/vKXTXK3xjr0sbGxnD6XESEEFixYgHvuuQc/+MEPTDcl2elqHPDh9D3G43G88sor6nfjYBC7v8H69esBwJXMLsO4gQ37PMGLLlPpzRpDG0ZD1djYCE3TlGEnIixfvhzt7e2Ix+O47rrr0rTQn3zySQwPD2PPnj1Zh2BImpub0dzcjP7+fqxatQoPP/ywqxvDihUrQET49a9/ndYxKoTApk2blHH90pe+pM6/ePFidSOTn3Xbtm2O1zGqTwoh0NbWhvr6+rS/QU9Pj2l0nzT+DFMobNjnCYVOypHGimi67V12qV5//fXKMB09ehS7du3C5s2bTZ59PB7HXXfdlWa4iUjFtn/v934Pv/vd79R7CxcuxPnz520N9vDwMFpbW5FIJDA0NOS6rf/9998HEanBFcZQyf3334/vf//7AKZvVvfeey8mJiZw9OhRtLW1qdi9mxujNbxi9PKNf4P+/n6eXsTMCmzY5wmFzia1Oz4Wiynd9PHxccRiMdTU1ChDOzk5if379+Oxxx4z6atL/H4/ent7bce+XXjhhVi/fj1+8pOfpI2pe/PNN5XnmwvJZFLdKNasWYN169aht7cXS5cuRTgcVvsZP9fk5CSi0ah6EiFKF+iy4jTqz/odtrS04Pjx4wXNi2UYO9iwzxMKnU1qd7xdpYdVB2bv3r3jAgyjAAAbEElEQVS2BnjZsmWora3FgQMHbK/3wQcfOI6+O3/+vPJ05eCJTF67pmkqP0BEJiM6MjKCkydP4rnnnnP0wonI9LShaRpGR0cRj8czGnfre3bfYXV19ayPKGTmIW5qIr3euI69vHCqb5fdkkIICgQCNDg46Dgow7rJxiBrLfjChQuzHtfZ2Zk2dGLFihWmjk/jzytWrCBd15UGjWwYcmrTt3aBGjchBOm6zh2gTFEA67EzuWInCwvAMekqNd3lMbFYDHv37s16Hal0ODU1ha985SuqwkXTNJw/fz7jsalUCg899BAuvvhipXWeSqXSShaFECr0YnwvmUwikUgAMGucCyEQCoXUd9DR0YFHHnkEyWTS5PFrmqaqZTguzpQqbNgZAJ9VzRgHOQSDQaxfvz5jgk/+vHr1asdkpxG/3w9N0zA1NQWfz2eqSZfeRjaIyHEohhACwWAQN910Ew4cOGB7vv379yMUCqG9vR09PT24++67kUwmsXXrVnVDAKD+m0qlcNttt6Gurk7V6nNcnCllPDHsQog1AB4DoAPYS0Tf8+K8zNxh7IAEPhv+DCAtaWqtxZbHZjPKsvSvtbUVAwMDGB0dNWmQuzHqmc7t8/lQU1ODtrY2ANMG3Pi+fFIYGhrC0NAQ3n33XYyNjSkDbpfElVRVVakSR46LMyWPm3hNpg3TxvxdAF8EEADwOoCVmY7hGHvpkUmzxKqzbpTnjUQiFI1Gs8bGhRBpMWk7TZlMm9O8ULleORBaCoEZhb3q6urS9GA0TUsT8pIaMFJ73pg/8Pr7ZkEvJlcwhzH2OgDvENG/AoAQ4h8AfB3AmxmPYkoKqzSAdfSabLDp6OhQLfvJZBLRaBS6ruPP//zP8ctf/hInT55EKpWCz+dTYQ2fz4fm5mZUVVWp68lzyXp3eT4n/H4/7rjjDjz55JP4+OOP095PpVLqaUM+afh8PvXz66+/jm984xsYGhpSxxBRWkz/tttuQ1VVFZ5++mn1OXbu3OmpZx6Px9Pq/9nzZ7zEC8N+CYBfG35/H8B/sO4khGgH0A4Ay5cv9+CyTCHE43FVrmhtZbcbvbZ7924VizZCMw1LzzzzDDRNg8/nw9q1a9Oud+jQISSTSezduxdXX301FixYoGLyuq6bkqgAVBnjtddei5UrVzoOoLbD5/OhtbUVZ86cUeGYqakpLF68GJ2dnXj44YfVDeWKK64wGfvm5mYkEgk1OFsIoZKtXmFX/8+GnfGSOUueEtFuALuB6WHWc3VdJp14PI7GxkblzT799NN4/PHHlYCXpmnYtWuXiiWHQiFs2bIla8t/KpXC5OQknn322Yx15UZDCkxXwxiTqHIQdTKZxE9/+lNcdNFFeO2119LO4zSrdMOGDQCmbyYSn8+nnkC+9KUvYevWrUgmk/jhD3+ozqNpmnpSKaSZi2GKjReG/TcALjX8vmzmNWaWyEfMy3jMwMCAKVE4Pj6O3t5eVRGTSqVw9913Q9d11QRkNeqBQCCtLR9wX9liZPHixfjwww/V78abQjKZxP79++Hzpf9T1XUdn//853Hu3Dn1mhBCJWflmoUQaG5uVjIIiUTCFLqRg7OlES+0mSsbra2t6Ovrw+TkpEkgjGE8w00gPtOG6ZvDvwL4A3yWPP33mY7h5Gn+5DMizXpMNBpNa8CRDTzyd2OTj0xMysRiQ0ODSpoaZ3zO5iaEoJUrV9KSJUtMr1mTn52dneozyxmmfr+fAoGA6fNbv498EpmFJEA5ecrkA+YqeUpEU0KIrQCew3SFTB8R/bzQ8zL2ZBLzsnry8vfR0VHTMYlEQjXhSG10a8u83+9Xr1t1xzs6OvDyyy9DCIFLL700TcslE1IZ8sILL8xJYpeI8NZbb6myRfkP+JVXXkFnZydOnTqFlpYWtLe3m44BPpMFliWciUTCtUfu9HRUqFqmneQAw3iFJzF2IjoE4FDWHcuIQrTLZ/OaTvHfTJKwuq6rUIYx3NDW1obXXnvNJI4FALW1tejp6QGAtCqZgYEBk+Z6rkZd1pJbjbo1Xi61ZADg9OnT6gZk3AeYroAZGxvDc889h3g8ju7ubrVO+bloJkErhDB9/mx/10zGu1C1TIaZVdy49V5vpR6KKcZE+EzXtD622z3GW3VPmpqaTL9HIpG0c8hQhXV+p6zblnNOZRhDzip1midayGZXTy5r6bOFfAKBgG14pdBwS1dXl6muX2rJZPt7McxsAdaKyZ/Z9sbsPHOnazp5jdb1ZJOEtZY0GkvuyOIFT01NIRaLYd++fSaZgGQyiQMHDsDn82WdeqRpmvLOM+0HTD9FfO9738PIyAh6e3tx4sQJ9UQwPj6ORCKB++67Dw899JA6l1HRMZlMpmmb5xJucSIUCpk6cUOhkHpvthOsDFMIbNhtmM1yNydD7XRNtzcZLyVhZcPQ+Ph4mlGmmVj1Nddck3F2Kc2EPzZu3GiqJ7fy+7//+7jjjjsATI/RSyQSOHHihGmfc+fOYefOnaq2fe3atWhubjZptthpmxcax04kEuoGIkshjXCcnClV2LDbMJvemJOhdjucIRQKqTiynd63VaDLae01NTWqnNE4oFnTNDzxxBOorq5Gb2+vbT16IBBQU4WchL+ISJVUNjc3Oxr2s2fPIhqN4umnn0ZzczOA6Zpz2SBERHj00UdVeaKu66irq0N7e7upzj6RSKgEb65/M6fcRmNjI4LBINezM+WHm3iN11upx9hnk3zLFbu6utLixvnGdY26MFKf3KoH09XVRevWrTOVQAKf6aHL81g1WaybjO0b92loaKAVK1Y4HuP3++nKK69MK7d0yj9k+04ylRZmO57LEplSAhxjL03cPg1Yvcj6+np0d3e7CsvYeaDWBiWp5Chb5uU1du/eja1bt6r5psbyQmA61vzII4/g7bffRlVVlfL87Tx7XddRU1OD6upqk+e7bt06bN682fE7mpycxNtvv62uKfVa7Lxxuycg+XomPXn5fQwNDamnDidZYg63MGWHG+vv9TafPXY3OHmRbr1T6z5WRcZwOKymFxn3iUQithUvskLF+rp8z+jVL1u2jNatW6fUEe0UIru6utKeBIybruvqekIIikQijt+TsWrHrhrG+JnklCSrkqXcgsEge+ZMSQP22MuXXOPw2Y4FoF4ztuhv3LhRtbM7DcqQAzfuuecePProo5iamjK9Lz11OWXogQceQCKRUElWuYZt27aZ1rtgwQKlEglM17Ffc801uPrqq1FTU2NKjNbU1KTlFYxJaPlZampq0N/fr+rsnfTkjU8sxutv2LCBvXOmImDDXoJkqsrJFhowHuvz+TA6OoqamhoEAgGTIZ2amsKZM2dMnalGo67rOr761a8iGAyisbERY2NjuPXWWwEABw8eTNOOkcnNLVu24Jvf/KZp/efOncNNN92kOkPlDWrHjh0qqUpEaGtrU52jxsSo0cjLUIrxBiaRImayikWWeUrtGOONQY7EM06LYs0WpmJw49Z7vXEoJjv5Ju1keCIcDqeFKKyDJuRQZpmY1DRNNSE5NQQFg8G0ZiLrJhOykUgkbV85SJqIKBKJpCVarWQaOO0UctE0jZqamjJ+d8aENCdHmXIBHIopb/JJ2hnDE9ahy7IcUA54kNK4qVRKKTgak5QHDhywvcbExETWgdNTU1M4fPgwnnvuOdNTAgD09/eb9Fyy4fT0Yg1LAcC+ffvUftu3b8/4/XFSlKlo3Fh/rzf22L1ncHCQmpqaTC3wPp8vLdEajUapqamJOjs7aeHChaYkpjG5GAwGHT32aDSascRRXt9uH6PHPjg4SIFAIOv4ObdPL1yayFQ6cOmxC7Iky+aC2tpasnYXMvkjPXVrzNjasGPcT9M0fPOb38TY2Bh6e3sxNTVlGtNmnJjk9/txyy23AIAqcfzrv/5rnD17NuO6ZAerEAJXX321KYZuXHsujWDFEGdjmFJBCHGSiGqz7cehmFlgro2PtS7dqM4Yi8XUCLwBgzJjKpXCD37wA9x3331KBdGYiDS2z6dSKVRVVWHfvn0mZUcrspNVsnbtWtTV1WX8HnIJiRQqlcsw8wY3br3XWyWHYuZC9c9O7dEYOpFqh3av+Xw+9ZpdLblMYGZKTtptuq5TZ2enq9BKvjglUhlmvgCXoRhtrm8klY70ipPJJMbHx1UdeSakjng8Hne17+rVq/HAAw9g9erViMfjqK+vx4YNG9QQC6PaoWRychKJRAK7du2C3+9XyVMn6uvr0dPTg9WrV6Onpwetra0IBALQNPt/MslkEmNjY9i5cyduvPFG7Ny503NvWiZSdV1n7RaGyQCHYjwmk9SrHZnCC7nI+xrnaOq6jpaWFrz00ktKmlfTNIRCIRXjfuihh/DOO++Y1iKHccgbjKwfP378OI4cOYIjR45g+/bteOGFF2xvCmfOnFG15EePHgUAxwqYfMJVLJXLMC5x49Z7vVVyKCbTcAan/TPVaRuFuoyv28kNyNCLFOoaHBxUgzHk7E9ZDWNs2ZdzTI1173at+MbrW8M4cpiHsRLG5/PlJbzFMIw9mItQjBDiz4QQPxdCpIQQWTO18wEp9arruuratEOGX0KhkG14IRaL4fz580ilUpicnMSWLVtU2OXIkSN48MEHbUe1AdNPCjt27MDIyAjq6urUH3tychIPP/ywqXrmxhtvxPHjx7FmzRolOWBsxbeuS15/06ZNCAaD0DQNPp8PTz75JFpbW1UljFyHXSjKSfaAYRhvKDQU8waAOwBEPVhLReAmXGANv9xzzz04deoUVq1ahYGBAYyMjKCvr88U7pBGUlaRWAcrDw0NpYVH+vv7sX37dpP6YiqVUvrr1kYeYyOQsRU/FAop42u8vl2r/hNPPIEtW7YglUo53thmc5AJwzDwJhQDYABArdv9KzkU4wZj+EU2Elkbi6zNPT6fz9TcI8nUTCT37+zsNL3e0NBAkUjElfZ4Ifrxblr6Wf+cYdwDlhTwhtmoSTd6rMbWfwBqapCcGSqrUFKpFDo6OgDA1HRkDMFILr/8ctx///1ob29HPB7HqVOnTJrqx44dw+DgIGpqarJqjzslazPhpjbdbh+uU2cYj8hm+QG8iOmQi3X7OuXgsQNoB3ACwInly5fPze2tQAqdzpPt3MapSFYPXSY8jdrkdjIBg4ODJsEuY/24VXfcmvD0+/1pUgPWp4K5THRynTrDZAZeeexEdINHN5DdAHYD05ICXpxztsnmrcppQ8lkEsFgMCcP0+ixVldXm8oIhRDKc5e/y012icoa+cbGRvWepmnYuXMnAKC7uxujo6MqUSr1zk+ePKm6Q5PJpIrpb9q0CQDw/PPP491338XixYvVU8FclRhy7J1hvIFDMRnIZGji8Ti2bNmiBk9IQ5uPIuPAwABaWlpw/PhxTExMQNd1NVxaDnYmImXsgc9q5AcGBpSxB4Dh4WFVf65pmgrxEE3rnbe1tZluRo2Njdi+fbtpTQ8//LBKrsqb1VyERLhOnWG8oSDDLoS4HcBOAEsA/B8hxCkiusmTlZUAmQzNwMCASTNF1/WcPUxrTNko2iWvMTQ0ZBpGIdE0Te0rh0ZomoYzZ86opwzpqdNMzD6RSGDbtm1qiIX8TC0tLXj++efVueVNxG1M3fh5CjXKLKfLMIVTkGEnoh8D+LFHaylJnAyNrFeXBvWJJ57I2SBZQz3S8BqvbTf0WQihvG3Z+r9lyxYkk0kcOnRIdZBKz192o8ouWGPtO/BZd2h/fz9WrVqFnTt35hwO4cQnw5QOHIrJE7tBD9a5nNlwE1NubW1Fb28vJicnAUy3/UsJXHnN0dFR5WUnk0ls3LgRy5cvR2NjI0ZGRpTR7+joQHV1NQCkGeH29nZl4MPhcM6edz7VMwzDzA5s2AtAevN23urIyAj6+/vVnE+n448cOYJYLIYzZ84oeV1rCeJLL72k9jl06BBOnjyJ119/HUTTUru6risvXTYXGb1ya2gFQEYjnE84hBOfDFM6sGH3AKu3ahzSLGPXmcbB9fX1qVr0p59+Wg27AD6LW8suTzlI2pgUBWDy0uXNRnaN2hlcr40wJz4ZpnRgw+4BVm/19OnTpvczzfkcGBhQYRYAJg/aLrkqr+Pz+ZTHbvXSMyVl5T6zYYQ58ckwpQEbdhdkq/aweqsjIyMYGhpS73/uc59TAl7Gc8rwis/nU8Y9EAggFAqp2Lk1uWqN6xvXJddpd5wxKSvXzEaYYSoUN11MXm/lpBWTb/dpNBqluro68vv9GSV2MaMDEw6HKRKJqE5UXdcpEAhQMBh01fVpXGcwGDRJ8LLuCsNUBmCtGG/IVu3hVObX3t6ORCKhOj3Pnz+PWCxmq++STCZRV1eHbdu2obu7W10PSI+du1lnLscxDFN5zDvDnmsTTbZqD6mbTkSYmJhALBZT529sbITP51OdoX19fWhtbTU1FQGA3+9X5w2FQmpsnTV2nss63R7HMEwF4sat93orVijGGlaJRqOuBLwyScwaQyp+vz8tBBKJRJT4lnUSUSQSMcnnGkW7nGR681knwzCVATgUk44xXDE+Po4tW7YozzhTp6RTonFgYEBpxQghUFNTo0IvMmzT2tqKffv2pXn8mSRypRRAIpHI6fM5SeFyCSLDzC/mlWF30kHPt1PSGv5oa2vDyMiIyYjnUt/tFPbJ1zhzmz/DzE/mlWE3GtlQKKRUEJ2adHItc6yvr08T2JL7uTGoUvdFdqw6dbVymz/DMBlxE6/xeiuVcsdMMWmnMsfZjGPbXbOQ4RNzOSSDYZjZBxxjz04mT9rO2wWmxbOkouOuXbvSOkoLiWnbXbMQDRZu82eY+cm8NuyZsDOoAwMDaiJRKpXC1q1blVqiXXjnyJEj6j03htXumvkaZ+MNxtp1yjBMhePGrfd6K5VQTDasYZfBwUHy+XyqvFHTNIpEIirc4ff71XxRXddN77kNhXgR6uEQDMNUJnAZitGKe1vJjXg8ju7ubsTj8Tm5Xn19PbZt22ZKgu7atQt+vx+apiEYDAL4TAI3mUxC0zTouo5AIGB6zxjOsUN+NgCma+aDUxiJYZj5QdmEYkqldK+9vd1U+QLAVKduHW9nV8NuxevPxtroDDO/KXTm6UMA1gKYAPAugA1EdM6LhVkpZumeNSFqTbpmioG7iY97/dk4acow8xw38RqnDUATAN/Mz98H8H03x+UTYy9W3HgurssxcYZh3IC5KHckoucNv/4MwH8s5HyZKJYXOhdPCuxhMwzjJV7G2O8E8L88PF8axRgOMVfxah58wTCMV2Q17EKIFwFU2bz1XSI6MLPPdwFMAXgmw3naAbQDwPLly/Na7GyRqamIvWmGYcoNMR22KeAEQvwlgE0AVhPRJ26Oqa2tpRMnThR0Xa8olWob65r4RsIwjBUhxEkiqs22X6FVMWsAdAL4mlujXmqUmlBWKd5oGIYpLwptUHoCwIUAXhBCnBJCPOXBmuYUGUOXTUXFrvnm5iKGYQql0KqYy71aSLHIFEMvRkiEm4sYhimUgmPs+VBKMXYnZjskkummwTF2hmHsmJMYeyUzm7H3bDcNLn1kGKYQykoEbC6Zzdg7x9EZhplN2GN3YDbr1zmOzjDMbFL2MfZyjUeX67oZhike8yLGXs413xxHZxhmtijrGDvHqhmGYdIpa8Neas1FDMMwpUBZh2JYoIthGCadsjbsAMeqGYZhrJR1KIZhGIZJhw07wzBMhcGGnWEYpsJgw84wDFNhsGFnGIapMNiw2xCPx9Hd3Y14PF7spTAMw+RM2Zc7ek05yxQwDMMA7LGnwTIFDMOUOwUZdiHEg0KIf56Zd/q8EGKpVwsrFixTwDBMuVOQbK8QYhERjc38fC+AlUQUyXZcqY/GY0ldhmFKkTmR7ZVGfYYLAMy9uPsswDIFDMOUMwUnT4UQ/x1AK4D/B+D6glfEMAzDFETWGLsQ4kUhxBs229cBgIi+S0SXAngGwNYM52kXQpwQQpw4e/asd5+AYRiGMeHZaDwhxHIAh4joq9n2LfUYO8MwTCniNsZeaFXMFYZfvw7gXwo5H8MwDFM4hcbYvyeE+AqAFID3AGStiGEYhmFml0KrYlq8WgjDMAzjDZ7F2HO6qBBnMe3h58rFAD70eDlzAa97buF1zy287rnjMiJakm2nohj2fBFCnHCTOCg1eN1zC697buF1lx6sFcMwDFNhsGFnGIapMMrNsO8u9gLyhNc9t/C65xZed4lRVjF2hmEYJjvl5rEzDMMwWSg7w16uGvBCiIeEEP8ys/YfCyEWF3tNbhBC/JkQ4udCiJQQouQrCIQQa4QQvxBCvCOE+E6x1+MGIUSfEOIDIcQbxV5LLgghLhVCHBVCvDnzb+QbxV6TG4QQC4QQQ0KI12fW/d+KvSavKbtQTL4a8MVGCNEE4J+IaEoI8X0AIKJvF3lZWRFCXInpzuIogG8RUcmK/AghdABvA7gRwPsAXgXwn4jozaIuLAtCiAYAHwOIudFaKhWEEF8A8AUiek0IcSGAkwDCZfB9CwAXENHHQgg/gJcBfIOIflbkpXlG2Xns5aoBT0TPE9HUzK8/A7CsmOtxCxG9RUS/KPY6XFIH4B0i+lcimgDwD5jWMCppiOgYgN8Vex25QkS/JaLXZn7+CMBbAC4p7qqyQ9N8PPOrf2YrCzvilrIz7MC0BrwQ4tcA1gH4r8VeTx7cCeBwsRdRgVwC4NeG399HGRiaSkAIsQJADYBXirsSdwghdCHEKQAfAHiBiMpi3W4pScPulQb8XJNt3TP7fBfAFKbXXhK4WTfDOCGE+DyAfgAdlifqkoWIkkS0CtNPznVCiLIJgbmh4AlKswER3eBy12cAHALwN7O4HNdkW7cQ4i8B3ApgNZVQciOH77vU+Q2ASw2/L5t5jZklZmLU/QCeIaIfFXs9uUJE54QQRwGsAVBWyetMlKTHnoly1YAXQqwB0AngNiL6pNjrqVBeBXCFEOIPhBABAH8B4Nkir6limUlC9gJ4i4geLfZ63CKEWCKr0oQQCzGdbC8LO+KWcqyK6Qdg0oAnopL3yoQQ7wAIAkjMvPSzMqnmuR3ATgBLAJwDcIqIbiruqpwRQtwMoAeADqCPiP57kZeUFSHE3wNoxLTa4P8F8DdE1FvURblACPEnAI4DGMH0/48A8F+I6FDxVpUdIcRVAPZh+t+IBuCHRPS3xV2Vt5SdYWcYhmEyU3ahGIZhGCYzbNgZhmEqDDbsDMMwFQYbdoZhmAqDDTvDMEyFwYadYRimwmDDzjAMU2GwYWcYhqkw/j99ljoOlc9WLgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dim = 2\n",
    "num_obs = 1000\n",
    "\n",
    "mean_true = np.random.random(dim)\n",
    "cov_true = np.random.random((dim, dim))\n",
    "cov_true = 0.1 * np.eye(dim) + np.full((dim, dim), 1.0)\n",
    "\n",
    "x = np.random.multivariate_normal(mean=mean_true, cov=cov_true, size=(num_obs, ))\n",
    "plt.plot(x[:, 0], x[:, 1], 'k.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define ``paragami`` patterns and a loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mvn_pattern = paragami.PatternDict(free_default=True)\n",
    "mvn_pattern['mean'] = paragami.NumericVectorPattern(length=dim)\n",
    "mvn_pattern['cov'] = paragami.PSDSymmetricMatrixPattern(size=dim)\n",
    "\n",
    "# ``example_utils.get_normal_log_prob`` returns the log probability of\n",
    "# each datapoint x up to a constant.\n",
    "def get_loss(x, sigma, mu):\n",
    "    return -1 * np.sum(\n",
    "        example_utils.get_normal_log_prob(x, sigma, mu))\n",
    "       \n",
    "get_free_loss = paragami.FlattenFunctionInput(\n",
    "    lambda mvn_par: get_loss(x=x, sigma=mvn_par['cov'], mu=mvn_par['mean']),\n",
    "    patterns=mvn_pattern,\n",
    "    free=True)\n",
    "\n",
    "mvn_par = mvn_pattern.random()\n",
    "mvn_free = mvn_pattern.flatten(mvn_par)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimization objectives."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use ``get_free_loss`` to define an optimization objective and pass it to ``scipy.optimize.minimize``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "objective = paragami.OptimizationObjective(get_free_loss)\n",
    "\n",
    "def Optimize(objective, x0=np.zeros(mvn_pattern.flat_length())):\n",
    "    opt_result = sp.optimize.minimize(\n",
    "        x0=x0,\n",
    "        fun=objective.f,\n",
    "        jac=objective.grad,\n",
    "        hessp=objective.hessian_vector_product,\n",
    "        method='trust-ncg')\n",
    "    return opt_result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Controlling printing.\n",
    "\n",
    "The frequency of printing can be controlled with ``set_print_every()``, and the count can be reset with ``reset()``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Optimizing:\n",
      "Iter 0: f = 1290.22412134\n",
      "Iter 1: f = 539.49481319\n",
      "Iter 2: f = 449.15367362\n",
      "Iter 3: f = 272.45823141\n",
      "Iter 4: f = 255.05895262\n",
      "Iter 5: f = 254.63015576\n",
      "Iter 6: f = 254.47898697\n",
      "Iter 7: f = 254.47819241\n",
      "Iter 8: f = 254.47818845\n",
      "Iter 9: f = 254.47818845\n",
      "Optimization terminated successfully.\n",
      "\n",
      "Optimizing printing less:\n",
      "Iter 0: f = 1290.22412134\n",
      "Iter 5: f = 254.63015576\n",
      "Optimization terminated successfully.\n",
      "\n",
      "Optimizing without reset:\n",
      "Iter 10: f = 1290.22412134\n",
      "Iter 15: f = 254.63015576\n",
      "Optimization terminated successfully.\n",
      "\n",
      "Optimizing without printing:\n",
      "Optimization terminated successfully.\n"
     ]
    }
   ],
   "source": [
    "print('\\nOptimizing:')\n",
    "opt_result = Optimize(objective)\n",
    "print(opt_result.message)\n",
    "\n",
    "print('\\nOptimizing printing less:')\n",
    "objective.reset()\n",
    "objective.set_print_every(5)\n",
    "opt_result = Optimize(objective)\n",
    "print(opt_result.message)\n",
    "\n",
    "print('\\nOptimizing without reset:')\n",
    "opt_result = Optimize(objective)\n",
    "print(opt_result.message)\n",
    "\n",
    "print('\\nOptimizing without printing:')\n",
    "objective.set_print_every(0)\n",
    "opt_result = Optimize(objective)\n",
    "print(opt_result.message)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessing derivatives.\n",
    "\n",
    "The function value and derivatives are methods of the optimization objective.  Under the hood, all the derivatives are calcualted with ``autograd``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f() evaluates the objective:\n",
      "254.47818844774238 = 254.47818844774238\n",
      "\n",
      "grad() evaluates the gradient:\n",
      "[-9.38462682e-07 -8.28401150e-07  1.12731491e-06 -9.35645267e-07\n",
      "  3.23708426e-06]\n",
      "\n",
      "hessian() evaluates the Hessian matrix:\n",
      "[[ 4.80152064e+03 -4.37310150e+03  9.83363903e-06 -6.91901344e-06\n",
      "  -1.47624300e-06]\n",
      " [-4.37310150e+03  4.90797543e+03 -8.10149773e-06  8.64973804e-06\n",
      "   1.65680224e-06]\n",
      " [ 9.83363896e-06 -8.10149785e-06  6.30553475e+03 -4.59689665e+03\n",
      "  -1.75268882e-06]\n",
      " [-6.91901349e-06  8.64973810e-06 -4.59689665e+03  4.90797543e+03\n",
      "   1.87129286e-06]\n",
      " [-1.47624310e-06  1.65680237e-06 -1.75268281e-06  1.87129159e-06\n",
      "   1.99999999e+03]]\n",
      "\n",
      "hessian_vector_product() evaluates the Hessian-vector product:\n",
      "[-8.83365609e-04  3.82200615e-05  1.14093879e-02 -9.77427408e-03\n",
      "  6.47416850e-03]\n"
     ]
    }
   ],
   "source": [
    "opt_x = opt_result.x\n",
    "\n",
    "print('f() evaluates the objective:\\n{} = {}'.format(\n",
    "    objective.f(opt_x), get_free_loss(opt_x)))\n",
    "\n",
    "grad = objective.grad(opt_x)\n",
    "print('\\ngrad() evaluates the gradient:\\n{}'.format(grad))\n",
    "\n",
    "print('\\nhessian() evaluates the Hessian matrix:\\n{}'.format(objective.hessian(opt_x)))\n",
    "\n",
    "print('\\nhessian_vector_product() evaluates the Hessian-vector product:\\n{}'.format(\n",
    "    objective.hessian_vector_product(opt_x, grad)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logging.\n",
    "\n",
    "By default, an optimization objective does no logging.  But by using ``set_log_every`` with a number larger than ``0``, a log of the values is kept."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Optimizing with logging:\n",
      "Optimization terminated successfully.\n"
     ]
    }
   ],
   "source": [
    "print('\\nOptimizing with logging:')\n",
    "objective.reset()\n",
    "objective.set_print_every(0)\n",
    "objective.set_log_every(2)\n",
    "opt_result = Optimize(objective)\n",
    "print(opt_result.message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The log is contained in the attribute ``optimization_log``.  By default, each entry in the log is a tuple containing the iteration number, argument, and function value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0, array([0., 0., 0., 0., 0.]), 1290.2241213426291)\n",
      "(2, array([ 0.09798938,  0.57638205,  0.10194361,  0.92848007, -1.16334881]), 449.1536736241908)\n",
      "(4, array([ 0.10983197,  0.58725075,  0.05844622,  0.94244548, -0.80874364]), 255.05895261654913)\n",
      "(6, array([ 0.13464741,  0.61326864,  0.04907768,  0.9356453 , -0.7953923 ]), 254.47898697444145)\n",
      "(8, array([ 0.13480492,  0.61343579,  0.04990915,  0.93661793, -0.79543076]), 254.4781884477883)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'Difference from optimum')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEKCAYAAAAb7IIBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl8V/Wd7/HXJwlhCTsJWwKEJLhQ64IpggoEXFqnjnpbuzh1pNZRCmg749yZduY+7u3tnT7mtvfOnXaqgiKo2Fq6aK3WOq1WVgVBNhFxIYQtYQv7JkuSz/3j942GGEzIL8n5Le/n4/F75Jzv7/zO+fwi/t455/zO+Zi7IyIi6Scj6gJERCQaCgARkTSlABARSVMKABGRNKUAEBFJUwoAEZE0pQAQEUlTCgARkTSlABARSVNZURfwSXJzc72wsDDqMkREksqqVav2untec8sldAAUFhaycuXKqMsQEUkqZra1JcvpEJCISJpSAIiIpCkFgIhImlIAiIikKQWAiEiaUgCIiKQpBYCISJpKyQDYffgE//LCBg4ePxV1KSIiCSslA+DA8VPMeXUzc5e26FoIEZG0lJIBcMHAnlx7YX8eX7qZYydroi5HRCQhpWQAAEwtK+Hg8dPMW7Et6lJERBJSswFgZo+Z2R4zW99grK+ZvWxmG8PPPmHczOynZlZuZuvMbFSD10wOy280s8nt83Y+cvmwPowp6svsJZs5WVPb3psTEUk6LdkDeAL4XKOx7wKvuPsI4JUwD3ADMCI87gFmQiwwgO8BVwCjge/Vh0Z7mlZWwq7DJ/jdmqr23pSISNJpNgDcfTGwv9HwzcDcMD0XuKXB+JMe8zrQ28wGAZ8FXnb3/e5+AHiZj4dKmxs3IpeL8nvy8KIKauu8vTcnIpJUWnsOYIC77wzTu4ABYTof2N5gucowdrbxdmVmTC8rYfPeY/zn+p3Nv0BEJI3EfRLY3R1osz+vzeweM1tpZiurq6vjXt9nPzWQorwcZizYRKxUERGB1gfA7nBoh/BzTxivAoY0WK4gjJ1t/GPcfZa7l7p7aV5esw1tmpWRYXxzQjEbdh5m4fvxB4qISKpobQA8D9R/k2cy8FyD8TvCt4HGAIfCoaI/AdebWZ9w8vf6MNYhbrk0n8G9ujBzwaaO2qSISMJryddA5wHLgPPNrNLM7gJ+CFxnZhuBa8M8wItABVAOPApMA3D3/cC/AG+Ex/8KYx0iOyuDu8cXsWLLft7Y0mGbFRFJaJbIx8VLS0u9rXoCf3Cqlqt+NJ9LCnrx+J2j22SdIiKJyMxWuXtpc8ul7JXAjXXNzuQbVxWy4L1q3t5xKOpyREQilzYBAPDXYwvp3jmLmQt1LkBEJK0CoFfXTtw+ZhgvvrWTLXuPRV2OiEik0ioAAL5xdSFZmRk8slh7ASKS3tIuAPr36MKXSwt4elUluw6diLocEZHIpF0AAEwZX0ydw+wlFVGXIiISmbQMgCF9u3HTJYP5xYptHDimtpEikp7SMgAAppYVc/xULU8s3RJ1KSIikUjbADhvQA+uGzmAJ5ZuUdtIEUlLaRsAANPKijn0gdpGikh6SusAuGxoH8YW9ePRJRVqGykiaSetAwBg+sQSdh8+yW9Xq22kiKSXtA+Aq0r6cXFBLx5etIma2rqoyxER6TBpHwBmxrSyErbuO86L63dFXY6ISIdJ+wAAuH7kAEr6d2fGgnK1jRSRtKEA4KO2ke/uOsKC9/Y0/wIRkRSgAAhuvnQw+b27MkNtI0UkTSgAgk6ZGdwzvoiVWw+wYrPaRopI6lMANPDl0iH0y8nmoQXlUZciItLuFAANdM3O5BtXD2fR+9Wsr1LbSBFJbQqARv567DB6dM5i5iKdCxCR1KYAaKRnl07cPjbWNrKi+mjU5YiItBsFQBO+cdVwsjMzeGSRGsaISOpSADQhr0dnvvKZIfx2TSU7D30QdTkiIu1CAXAWd48ros7h0cWboy5FRKRdKADOYkjfbtx86WDmrdjGfrWNFJEUpAD4BFMnFPPBabWNFJHUpAD4BCMG9OD6kQN44rXNHFXbSBFJMQqAZkybWMLhEzX8YvnWqEsREWlTCoBmXDqkN1eV9OPRJZs5cVptI0UkdcQVAGb2d2b2tpmtN7N5ZtbFzIab2XIzKzezX5lZdli2c5gvD88XtsUb6AjTy0qoPnKSZ1ZXRl2KiEibaXUAmFk+8C2g1N0vAjKBrwI/An7s7iXAAeCu8JK7gANh/MdhuaQwtrgflwzpzSOLKtQ2UkRSRryHgLKArmaWBXQDdgKTgKfD83OBW8L0zWGe8Pw1ZmZxbr9DxNpGFrNt/3H+8NbOqMsREWkTrQ4Ad68C/g3YRuyD/xCwCjjo7vVfmakE8sN0PrA9vLYmLN+v8XrN7B4zW2lmK6urq1tbXpu77sIBjOjfnZkLN6ltpIikhHgOAfUh9lf9cGAwkAN8Lt6C3H2Wu5e6e2leXl68q2szGRnG1LJY28j576ptpIgkv3gOAV0LbHb3anc/DfwWuAroHQ4JARQAVWG6ChgCEJ7vBeyLY/sd7i8vibWNfEjN40UkBcQTANuAMWbWLRzLvwbYACwAbg3LTAaeC9PPh3nC8/M9yT5FO2Vm8M0JRazedpDlahspIkkunnMAy4mdzF0NvBXWNQv4DnC/mZUTO8Y/J7xkDtAvjN8PfDeOuiPzpdIh5HbPZsZCNYwRkeSW1fwiZ+fu3wO+12i4AhjdxLIngC/Fs71E0KVTrG3k//nje7xVeYhPF/SKuiQRkVbRlcCtcPuYYfToksXMRWoeLyLJSwHQCj27dOKOscP4z/W72KS2kSKSpBQArXRnaBv5sM4FiEiSUgC0Um73ztw2eijPrqmi6qDaRopI8lEAxOHu8UUAPLpYzeNFJPkoAOKQ37srt1yWzy/f2Ma+oyejLkdE5JwoAOL0zQnFnKypU9tIEUk6CoA4lfTvzmdHDuSJpVs4cuJ01OWIiLSYAqANTJtYzJETNTy1fFvUpYiItJgCoA1cXNCbcSNyma22kSKSRBQAbWRqWTF7j57kN6vUNlJEkoMCoI2MLerHZUN7M2vxJrWNFJGkoABoI7G2kSVs3/8BL6xT20gRSXwKgDZ0zQX9OW9Ad2YsLKeuLqlaHYhIGlIAtKGMjNhewPu7j/KK2kaKSIJTALSxGy8exJC+ahspIolPAdDGsjIzuGd8MWu3H2RZRVK1PBaRNKMAaAdfuryA3O6dmalbRYtIAlMAtIMunTL5m3HDWbJxL+sqD0ZdjohIkxQA7eRrVwylZ5csZizQXoCIJCYFQDvp0aUTk68s5E8bdlG+50jU5YiIfIwCoB19/cpCOmdlMHOhGsaISOJRALSjfqFt5HNr1TZSRBKPAqCd3T2uCDO1jRSRxKMAaGeDe3fllkvzmbdiG3vVNlJEEogCoAN8s6yYU7V1PP7a5qhLERH5kAKgAxTndeeGiwby5LKtHFbbSBFJEAqADjKtrIQjJ2r4+etboy5FRARQAHSYi/J7Mf68PB57VW0jRSQxxBUAZtbbzJ42s3fN7B0zG2tmfc3sZTPbGH72Ccuamf3UzMrNbJ2ZjWqbt5A8ppUVs/foKX6zcnvUpYiIxL0H8B/AH939AuAS4B3gu8Ar7j4CeCXMA9wAjAiPe4CZcW476VwxvC+jhvbm4UUVnFbbSBGJWKsDwMx6AeOBOQDufsrdDwI3A3PDYnOBW8L0zcCTHvM60NvMBrW68iRkZkyfWELVwQ/4/Zs7oi5HRNJcPHsAw4Fq4HEzW2Nms80sBxjg7vVNcXcBA8J0PtDw2EdlGDuDmd1jZivNbGV1dXUc5SWmSRf054KBPZi5cJPaRopIpOIJgCxgFDDT3S8DjvHR4R4APNYS65w+5dx9lruXuntpXl5eHOUlJjNjalkxG/cc5eV3dkddjoiksXgCoBKodPflYf5pYoGwu/7QTvhZ3xy3ChjS4PUFYSztfP7TgxjatxszFm5S20gRiUyrA8DddwHbzez8MHQNsAF4HpgcxiYDz4Xp54E7wreBxgCHGhwqSitZmRlMmVDEm9sPsnST2kaKSDTi/RbQfcBTZrYOuBT4V+CHwHVmthG4NswDvAhUAOXAo8C0OLed1L44qoD+PTozY2F51KWISJrKiufF7r4WKG3iqWuaWNaB6fFsL5XUt4381xffZe32g1w6pHfUJYlImtGVwBH6qyuG0atrJ2Ys0F6AiHQ8BUCEunfOYvKVhby0YTcbd6ttpIh0LAVAxO68spCunTKZuUjN40WkYykAItYnJzu0jdzB9v3Hoy5HRNKIAiAB3D1+OBkGjy5R20gR6TgKgAQwqFdXvnBZAb96YzvVR9Q2UkQ6hgIgQUyZUMSp2joeU9tIEekgCoAEUZTXnb/49CB+rraRItJBFAAJZOqEYo6crOFny9Q2UkTanwIggVyU34sJoW3kB6fUNlJE2pcCIMFMn1jCvmOn+LXaRopIO1MAJJjRw/tSOqwPsxarbaSItC8FQAKqbxv53Fq1jRSR9qMASEBl5+dx4aCezFxYrraRItJuFAAJqL5t5KbqY7y0YVfU5YhIilIAJKjPf3oQhf3UNlJE2o8CIEFlZhhTJhSzrvIQr5WrbaSItD0FQAL7wqh8BvTszENqGCMi7UABkMA6Z2Vy97gillXsY822A1GXIyIpRgGQ4G4bPZTe3ToxY6EaxohI21IAJLiczllMHlvIyxt2894utY0UkbajAEgCX7+ykG7ZmTystpEi0oYUAEmgT042fzV6KM+/qbaRItJ2FABJ4m/GFZFh8Mhi7QWISNtQACSJgb26cOvlBfx6ZSV7jpyIuhwRSQEKgCQyZXwxNbV1zHlVbSNFJH4KgCRSmJvD5y8ezFOvb+PQB2obKSLxUQAkmakTijl6soafLdsSdSkikuQUAElm5OCeTDw/j8de26K2kSISl7gDwMwyzWyNmb0Q5oeb2XIzKzezX5lZdhjvHObLw/OF8W47XU2fWML+Y6f45Rvboi5FRJJYW+wBfBt4p8H8j4Afu3sJcAC4K4zfBRwI4z8Oy0krlBb2ZXRhXx5dXMGpGrWNFJHWiSsAzKwA+DwwO8wbMAl4OiwyF7glTN8c5gnPXxOWl1aYOrGYHYdO8Lu1VVGXIiJJKt49gJ8A/wjU/xnaDzjo7jVhvhLID9P5wHaA8PyhsLy0Qtl5eYwc1JOHF22iVm0jRaQVWh0AZnYjsMfdV7VhPZjZPWa20sxWVldXt+WqU4qZMW1iMRXVx3jpbbWNFJFzF88ewFXATWa2BfglsUM//wH0NrOssEwBUH+MogoYAhCe7wV8rNWVu89y91J3L83Ly4ujvNR3w0WDGJ6bw0MLy9U2UkTOWasDwN3/yd0L3L0Q+Cow392/BiwAbg2LTQaeC9PPh3nC8/Ndn1pxycwwvjmhiPVVh1mycW/U5YhIkmmP6wC+A9xvZuXEjvHPCeNzgH5h/H7gu+2w7bTzXy4rYGDPLsxYqLaRInJusppfpHnuvhBYGKYrgNFNLHMC+FJbbE8+kp2Vwd+MG84P/vAOq7Ye4PJhfaIuSUSShK4ETgG3jR5Kn26dmKm9ABE5BwqAFJDTOYuvXzmcP7+zh3d3HY66HBFJEgqAFDH5ymHkZGcyU83jRaSFFAApone3bL42Zhi/f3MH2/apbaSINE8BkELuuno4WRkZPKy2kSLSAgqAFDKgZxduLS3g6ZWV7DmstpEi8skUAClmyvgiaurUNlJEmqcASDHD+uVw48WD+fnrWzl0XG0jReTsFAApaGpZMcdO1TJ32ZaoSxGRBKYASEEXDurJNRf05/HXNnP8VE3zLxCRtKQASFHTJhZz4Php5q3YHnUpIpKgFAAp6vJhfblieF9mL1HbSBFpmgIghU2bWMLOQyf43Rq1jRSRj1MApLDxI3L51OCezFTbSBFpggIghZkZ0yeWsHnvMf64Xm0jReRMCoAU99lPDaQoN4cZahspIo0oAFJcrG1kMW/vOMyi96ujLkdEEogCIA3cclk+g3p1YYZuFS0iDSgA0kB2VgZ3jytixeb9rNyyP+pyRCRBKADSxFdHD6FvTrb2AkTkQwqANNEtO4s7ryxk/rt72LBDbSNFRAGQVu4YWxhrG7lIewEiogBIK726deL2scP4w7odbNl7LOpyRCRiCoA0c9fVw8nKzOCRxRVRlyIiEVMApJn+PbrwpcsLeGZVJbvVNlIkrSkA0tCU8cXUujN7ifYCRNKZAiANDe3Xjb+8eBBPLd/GweOnoi5HRCKiAEhTU8tKOH6qlieWbom6FBGJiAIgTZ0/sAfXXjiAJ5Zu4dhJtY0USUetDgAzG2JmC8xsg5m9bWbfDuN9zexlM9sYfvYJ42ZmPzWzcjNbZ2aj2upNSOtMm1jMweOnmbdiW9SliEgE4tkDqAH+3t1HAmOA6WY2Evgu8Iq7jwBeCfMANwAjwuMeYGYc25Y2MGpoH8YU9eXRJRWcrKmNuhwR6WCtDgB33+nuq8P0EeAdIB+4GZgbFpsL3BKmbwae9JjXgd5mNqjVlUubmD6xhN2HT/LsarWNFEk3bXIOwMwKgcuA5cAAd98ZntoFDAjT+cD2Bi+rDGMSoatLcvl0fi8eVttIkbQTdwCYWXfgGeBv3f2Mu4x5rAXVOX2qmNk9ZrbSzFZWV6uBSXuLtY0sZsu+40x/ajXv7ToSdUki0kHiCgAz60Tsw/8pd/9tGN5df2gn/NwTxquAIQ1eXhDGzuDus9y91N1L8/Ly4ilPWuj6kQP51qQSlmys5rM/WcyUn61kfdWhqMsSkXYWz7eADJgDvOPu/97gqeeByWF6MvBcg/E7wreBxgCHGhwqkghlZBj3X38+r35nEt+aVMLSTfu48YFX+cYTb7B624GoyxORdmKtbRRuZlcDS4C3gLow/M/EzgP8GhgKbAW+7O77Q2A8CHwOOA7c6e4rP2kbpaWlvnLlJy4i7eDwidM8uXQLc17dzIHjp7m6JJd7J5Uwpqhf1KWJSAuY2Sp3L212udYGQEdQAETr2Mkanlq+lVmLN7P36ElGF/blvmtKuLokl1iei0giUgBImzlxupZfrtjGw4sq2HX4BJcO6c19k0qYdEF/BYFIAlIASJs7WVPLM6uqmLGwnMoDHzByUE/um1TCZz81kIwMBYFIolAASLs5XVvH79ZUMWPhJjbvPcaI/t25d1IJN148mEwFgUjkFADS7mrrnBfW7eDB+eVs3HOU4bk5TCsr5pbL8umUqfsMikRFASAdpq7O+dPbu3hgfjkbdh6moE9XppYVc+vlBXTOyoy6PJG0owCQDufuzH93Dz+dX86b2w8ysGcXpkwo4rbRQ+nSSUEg0lEUABIZd+fV8r088Eo5K7bsJ7d7Z+4eN5zbxwwjp3NW1OWJpDwFgCSE5RX7eGB+Oa+W76VPt07cdfVw7riykJ5dOkVdmkjKUgBIQlm97QAPzi9n/rt76NElizuvLOTOq4bTJyc76tJEUo4CQBLS+qpDPDi/nD++vYuc7ExuHzuMu8cVkdu9c9SliaQMBYAktPd2HeGhBeW8sG4H2VkZ3DZ6KFPGFzOwV5eoSxNJegoASQoV1Ud5aMEmfre2ikwzvlRawNSyYgr6dIu6NJGkpQCQpLJt33FmLtrE06u24w5fGJXPtLISCnNzoi5NJOkoACQp7Tj4AbMWVzBvxTZO19Zx0yWDuXdSCSX9e0RdmkjSUABIUttz5ASzl2zmZ8u2cqKmlhsuGsi9E0cwcnDPqEsTSXgKAEkJ+4+dYs6rFcxdupWjJ2u49sIB3DephEuG9I66NJGEpQCQlHLo+GmeWLqFx17bzKEPTjP+vDzum1TCZwr7Rl2aSMJRAEhKOnqyhp8t28rsJRXsO3aKMUV9+dakEYwt7qfmNCKBAkBS2vFTNcxbsZ1HFm1iz5GTjBram/uuGUHZeXkKAkl7CgBJCydO1/KblduZuXATOw6d4NP5vbh3UgnXXThAXcokbSkAJK2cqqnj2TWVPLRgE9v2H+eCgT24d1IJN1w0SF3KJO0oACQt1dTW8fvQpWxT9TGK83KYPrGEmy4ZTJa6lEmaUABIWqutc/64fhcPzN/Iu7uOMLRvN6aVFfOFUQVkZykIJLUpAESItav88zu7eXBBOesqDzG4Vxe+WVbMl0uHqEuZpCwFgEgD7s6i96t5YH45q7YeIK9HZ6aML+KvrhhKt2x1KZPUogAQaYK7s6xiHw+8Us6yin30zcmOdSkbO4we6lImKUIBINKMlVv288D8cha9X02vrp2486pC7rxyOL26KQgkuSkARFroze0HeXBBOS9v2E33zlncMXYYd109nH7qUiZJSgEgco427DjMQwvKeXH9TrpkZXL7mKHcPa6I/j3VpUySS8IGgJl9DvgPIBOY7e4/PNuyCgCJQvmeIzy0YBPPra0iKzOD2z4zhCkTihncu2vUpYm0SEIGgJllAu8D1wGVwBvAbe6+oanlFQASpS17jzFz4SaeWV2JGdx6eQFTJ5QwtJ/aVUpia2kAdPQVMaOBcnevcPdTwC+Bmzu4BpEWKczN4Ue3XszCfyjjq58ZyjOrqpj4/xZy/6/Xsqn6aNTlicSto78AnQ9sbzBfCVzRwTWInJOCPt34l1su4t5JJcxaXMFTy7fy7JoqchpcP3DG3Ybs42P1dyhteKPSJp8/Y+zjK23q9WeOfTRjn1DHGWu2j083tZ4zt2kfG+Mc3ps0r+z8PP7b50e26zYS7goYM7sHuAdg6NChEVcj8pEBPbvw328cydSyYn71xnb2HzsFQP1RVOejw6lNHVlteLjVzxg/++ubWq7hMw3Hzphu6vlPWK7hAmdu8+M1N73Opt8bTbw3aZkBHfDlg44OgCpgSIP5gjD2IXefBcyC2DmAjitNpGVyu3dm+sSSqMsQiVtHnwN4AxhhZsPNLBv4KvB8B9cgIiJ08B6Au9eY2b3An4h9DfQxd3+7I2sQEZGYDj8H4O4vAi929HZFRORMujG6iEiaUgCIiKQpBYCISJpSAIiIpCkFgIhImkro20GbWTWwNY5V5AJ726ictqS6zo3qOjeq69ykYl3D3D2vuYUSOgDiZWYrW3JHvI6mus6N6jo3quvcpHNdOgQkIpKmFAAiImkq1QNgVtQFnIXqOjeq69yornOTtnWl9DkAERE5u1TfAxARkbNIyQAws8+Z2XtmVm5m3426nnpm9piZ7TGz9VHXUs/MhpjZAjPbYGZvm9m3o64JwMy6mNkKM3sz1PX9qGtqyMwyzWyNmb0QdS31zGyLmb1lZmvNLGGaaZtZbzN72szeNbN3zGxsAtR0fvg91T8Om9nfRl0XgJn9Xfg3v97M5plZu3WGSblDQOfaeL4jmdl44CjwpLtfFHU9AGY2CBjk7qvNrAewCrgl6t+XxfoI5rj7UTPrBLwKfNvdX4+yrnpmdj9QCvR09xujrgdiAQCUuntCfafdzOYCS9x9dugD0s3dD0ZdV73wmVEFXOHu8Vx31Ba15BP7tz7S3T8ws18DL7r7E+2xvVTcA0jYxvPuvhjYH3UdDbn7TndfHaaPAO8Q690cKY+p77zeKTwS4q8VMysAPg/MjrqWRGdmvYDxwBwAdz+VSB/+wTXApqg//BvIArqaWRbQDdjRXhtKxQBoqvF85B9oycDMCoHLgOXRVhITDrOsBfYAL7t7QtQF/AT4R6Au6kIaceAlM1sVemsnguFANfB4OGQ228xyoi6qka8C86IuAsDdq4B/A7YBO4FD7v5Se20vFQNAWsHMugPPAH/r7oejrgfA3Wvd/VJivaNHm1nkh83M7EZgj7uvirqWJlzt7qOAG4Dp4ZBj1LKAUcBMd78MOAYk0nm5bOAm4DdR1wJgZn2IHbEYDgwGcszs9vbaXioGQLON5+VM4Rj7M8BT7v7bqOtpLBwyWAB8LupagKuAm8Lx9l8Ck8zs59GWFBP+esTd9wDPEjscGrVKoLLB3tvTxAIhUdwArHb33VEXElwLbHb3anc/DfwWuLK9NpaKAaDG8+cgnGydA7zj7v8edT31zCzPzHqH6a7ETuq/G21V4O7/5O4F7l5I7N/WfHdvt7/QWsrMcsJJfMIhluuByL9t5u67gO1mdn4YugaI/AsZDdxGghz+CbYBY8ysW/h/8xpi5+XaRYf3BG5vidx43szmAWVArplVAt9z9znRVsVVwF8Db4Xj7QD/HHo3R2kQMDd8QyMD+LW7J8xXLhPQAODZ2GcGWcAv3P2P0Zb0ofuAp8IfZBXAnRHXA3wYlNcBU6KupZ67Lzezp4HVQA2whna8IjjlvgYqIiItk4qHgEREpAUUACIiaUoBICKSphQAIiJpSgEgIpKmFAASFzOrDXdTfDvcufPvzSwjPFdqZj8N053N7M9h2a+Y2bjwmrXhe/4Jx8wuCPWtMbPiqOtpyMz+udH80qhqkeSlr4FKXMzsqLt3D9P9gV8Ar7n79xotNwb4gbtfG+YfBl519xZdRRsuijF377D774RbiWe5+w+irqWxhr93kVZzdz30aPUDONpovgjYBxixi95eAPoD5cAhYC2xC2/2A5uJ3X4C4B+IXcW9Dvh+GCsE3gOeBN4GhhG7wnUZsQtlfgN0D8tuAb4fxt8CLgjj3YHHw9g64IthvMn1NHgffwHsInYbkQVnqeW2sN71wI8a/k6A/xuW+zOxWzIsJHYR1E1N/A4tLL8+rO8rYbwMWAz8IWz7YWJ77T8EasPv8qmG/x3CaxYBz4Xt/RD4GrAirLs4LPcEcGvj/44tfb0eqfGIvAA9kvvROADC2EFiV6aWAS+EsQ+nw/yHH0Dhw3hW+CDMIBYa48OHbh0wJiyXGz4Qc8L8d4D/Eaa3APeF6WnA7DD9I+AnDbbb55PW0+h9/E/gv4bpxrUMJnbZfh6xK2/nE+ujALG7ct4Qpp8FXiJ2O+tLgLVNbOeLwMvErlwfENY7KPzOThAL1cywzK1N/d4bfYAfDK/vTCzA6gP12/W/i2YCoNnX65Eaj5S7FYQkpevDY02Y7w6MIPZBuNU/agIzBhgJvBZueZBN7K/4evU3slsFfCFMX0vsnj0AuPuBcEfPT1rP2TSs5TPAQnevBjCzp4iF1u+AU0D9bRjeAk66+2kze4sYQ6/CAAABx0lEQVRYkDR2NTDP3WuB3Wa2KKz/MLDC3SvCNuaFZZ9ups433H1neM0mYgFUX8vEFrzPeF8vSUIBIG3KzIqIHZ7YA1zY0pcB/9vdH2m0rkJitw9uuNzL7n7bWdZzMvys5ZP/bTe3nrM51vwiAJx29/qTa3X1dbl7XWjycS4an6RryUm7kw2m6xrM1/HR76WG8CWQcNI++xxfLylA3wKSNmNmecSOUz/Y4AOwJf4EfCP0JMDM8sMJ5cZeB64ys5KwXI6ZndfMul8GpjeosU8r19PYCmCCmeWGG9bdRuzYeWssAb4SGuDkEduTWBGeGx3ubJsBfIVYu0CA0+E23q21Bbg8TN9E7BCVpBkFgMSra/3XQImd8HyJ2MnYFvNYx6NfAMvCYZKngR5NLFcNfB2YZ2briB22uaCZ1f8A6GOxBttvAhNbuZ7Gtewk1thkAfAmsMrdnzuXdTTwLLET1G8SO5fwjx67jTLETow/SOyWwJvDshA7Z7IuHHpqjUeJBdibwFhavncjKURfAxVJUGZWRuwkdEI0nZfUoz0AEZE0pT0AEZE0pT0AEZE0pQAQEUlTCgARkTSlABARSVMKABGRNKUAEBFJU/8fKSbFqUA10C4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for log_entry in objective.optimization_log:\n",
    "    print(log_entry)\n",
    "\n",
    "plt.plot(\n",
    "    [ entry[0] for entry in objective.optimization_log],\n",
    "    [ entry[2] - opt_result.fun for entry in objective.optimization_log])\n",
    "plt.xlabel('Iteration')\n",
    "plt.xlabel('Difference from optimum')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method ``reset()`` clears the log as well as the iteration count.  Note that you can clear the log only using ``reset_log()`` and clear the iteration count only using ``reset_iteration_count()``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "objective.reset()\n",
    "print(objective.optimization_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overloading.\n",
    "\n",
    "Both the logging and progress methods can be overloaded to provide custom behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomObjective(paragami.OptimizationObjective):\n",
    "    def print_value(self, num_f_evals, x, f_val):\n",
    "        grad = objective.grad(x)\n",
    "        print('Iteration {}: gradient norm = {:0.8f}'.format(\n",
    "            num_f_evals, np.linalg.norm(grad)))\n",
    "\n",
    "    def log_value(self, num_f_evals, x, f_val):\n",
    "        grad = self.grad(x)\n",
    "        self.optimization_log.append((num_f_evals, np.linalg.norm(grad)))\n",
    "\n",
    "custom_objective = CustomObjective(get_free_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Optimizing with custom printing and logging:\n",
      "Iteration 0: gradient norm = 1325.81210303\n",
      "Iteration 1: gradient norm = 822.69307418\n",
      "Iteration 2: gradient norm = 1380.15778047\n",
      "Iteration 3: gradient norm = 343.45510333\n",
      "Iteration 4: gradient norm = 44.95265409\n",
      "Iteration 5: gradient norm = 14.74702496\n",
      "Iteration 6: gradient norm = 1.23658409\n",
      "Iteration 7: gradient norm = 0.09246396\n",
      "Iteration 8: gradient norm = 0.00029502\n",
      "Iteration 9: gradient norm = 0.00000377\n",
      "Optimization terminated successfully.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'Log gradient norm')"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEKCAYAAAAW8vJGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xd81fX9xfHXO5sECCNhB8KGABIhRRBworKEuqVoSxd14GirXbaV2mH9tVJFrYq7oqK1DmSIIoqgMsKegTDChrBJgIQkn98f96JJTCAQbr65uef5eNwH937v5957uEpOvvNjzjlEREROCvM6gIiIVC8qBhERKUHFICIiJagYRESkBBWDiIiUoGIQEZESVAwiIlKCikFEREpQMYiISAkRXgc4GwkJCS45OdnrGCIiQWXRokV7nXOJpxsXlMWQnJxMenq61zFERIKKmWVVZJw2JYmISAkqBhERKSGgxWBmHc1sabHbYTO7t9SYS8zsULExfwxkJhERObWA7mNwzmUAqQBmFg5sB94tY+gc59zQQGYREZGKqcpNSZcDG5xzFdr5ISIi3qjKYrgZeKOc5/qY2TIzm25mXaowk4iIlFIlxWBmUcAw4L9lPL0YaOWc6w48AbxXznuMNrN0M0vPzs4OXFgRkRBXVWsMg4DFzrndpZ9wzh12zuX4708DIs0soYxxE5xzac65tMTE056fUaa9OXkcOnrirF5bU+QVFLJ+9xE+XLmLZ2dv4LOMPRQVaXpXEflGVZ3gNoJyNiOZWRNgt3POmVkvfGW1LxAhnpyVyctfbqZebCStGsaR3DCWVg3jaJ0Q638cR/3YSMwsEB9fZZxz7M3JZ0N2Dhuzc9mYncPGvb4/t+w/SukeaNkglpEXtOSGtCQaxEV5E1pEqg1zLrC/LZpZHLAFaOOcO+RfdhuAc+4ZMxsD3A4UAMeAXzjnvjzVe6alpbmzOfN5UdYBFmcdYNO+XLL25bJ571F2HDpG8a+gTkwErRPiyiyOhnFR1ao0jp8oJGvf0a9/8G/Yk8MGfwEcOV7w9bjoiDDaJNamTWIcbRPiaJNYm7aJtUlqUIs56/fy6rwsFmzaT1REGEO7NWVk71b0aFmvWv1dRaTyzGyRcy7ttOMCXQyBcLbFUJa8gkK27j/G5r25bN6XS9a+o1//ue1Ayd+ua0dH0KphLMkN40gutpaR3DCWxDrRAflB6pwj+0geG7Jzv1kD2Ov7s3S+pvExtEmMo02CvwT8ZdAsvhZhYafOtm73ESbOy+KdxdvJySsgpWldbundiuGpzYiLDsorp4hIKSqGcyC/oIhtB46WKItNe31rG1sPHKOw2E/l2KjwEmsZyQ1jSU7wFUejOtGn/cF8/EQhm/bmltj0syE7h03ZuRzJ++a3/1qR4bROiCvxg79tYm1aJ8Sdkx/guXkFvLd0O69+lcXaXUeoEx3BtT2ac0vvVrRvXKfS7y8i3lExBNiJwiJ2HDzG5n1HydqX6y8MX4Fs3X+UE4XffK8xkWG0auBby0hu6NtMVegcG/Z8s+1/+8GSm7Saxcf4N/nEfb0ZqE1ibZrWjTltyZwLzjkWbznAq19lMW3FLvILi7igdQNu7dOKK1OaEBWhq6mIBBsVg4cKi5y/NHJ9xbE39+sCydp/lPyCIsC3ltE64Zvf/E8WQeuEOGKjqs/mm305ebyVvo3X5mex7cAxEmpHM6JXEiN6taRZvVpexxORClIxVFOFRY5dh48TZtCkbkxQ7eAtLHJ8vi6bifOymJWxBwMu79yYW3u3ol+7hCpZkxGRs1fRYqg+v5aGiPAwo3mQ/pYdHmZc2qkRl3ZqxNb9R3l9wRbeWriVj1fvplXDWG65oBXX92xBfR3yKhLUtMYglZJXUMiHK3cxcV4WCzcfICoijKvPa8YtvVuSmqRDXkWqE21Kkiq3dtdhJs7L4t3F28nNL6Rr87rcckErhqU2q1b7TERClYpBPJOTV8C7S7Yz8assMnYfoU5MBNf1aMEtvVvRrlFtr+OJhCwVg3jOOcfCzQeYOC+L6St3cqLQ0adNQ27t04orUhoTGa5DXkWqkopBqpXsI3m8lb6V1+dvYfvBYzSqE83NvVoyolcSTeODc2e8SLBRMUi1VFjk+CxjD6/Oy2L2umzCzBjQuRH3DuhA56Z1vY4nUqOpGKTa27LvKK8tyGLSgq3k5BXw/T6t+PkVHagbE+l1NJEaScUgQePg0Xz+MSOD1xdsIaF2NA8M7szw1GY61FXkHKtoMWjvn3iuXmwUf72mG+/d0Zem8THc++ZSbpowj4xdR7yOJhKSVAxSbXRPqse7d/Tlb9d0I2PXEQaPn8Nfp64mp9jVZUUk8FQMUq2Ehxnfu6Aln953CTf0bMFzczZx+aOf8cGyHQTjZk+RYBTwYjCzzWa2wsyWmtm3dgyYz3gzyzSz5WbWI9CZpPprEBfF3687j3fuuJDEOtHc9cYSRj4/n8w92rwkEmhVtcZwqXMutZydHoOA9v7baODpKsokQaBHy/q8f2c//jy8Cyu3H2LQ43P4+/S15GrzkkjAVIdNScOB/zifeUA9M2vqdSipPsLDjFv7JDPrvksYntqcZ2ZvYMC42UxbsVObl0QCoCqKwQEfmdkiMxtdxvPNga3FHm/zLxMpIaF2NP+8oTtv39aHerFR3PHaYr7/4gI2Zud4HU2kRqmKYujnnOuBb5PRnWZ20dm8iZmNNrN0M0vPzs4+twklqKQlN+CDMX0Ze3UKS7ccZOBjc/jHjLUcyy/0OppIjRDwYnDObff/uQd4F+hVash2IKnY4xb+ZaXfZ4JzLs05l5aYmBiouBIkIsLDGNW3NZ/cdzFDz2vKU5/6Ni/NWLVLm5dEKimgxWBmcWZW5+R94EpgZalhk4Hv+49O6g0ccs7tDGQuqTka1Ylh3E2pvDm6N7WjI/jZq4v44csLydqX63U0kaAV6DWGxsBcM1sGLACmOuc+NLPbzOw2/5hpwEYgE3gOuCPAmaQGuqBNQ6bc3Y/fD+lM+uYDXPGvzxn38TqOn9DmJZEzpWslSY2z+/Bx/jp1DZOX7SCpQS3GXt2Fyzs39jqWiOd0rSQJWY3rxjB+xPm8/tMLiI4I58evpPOTVxaydf9Rr6OJBAUVg9RYF7ZNYNrd/fnd4E58uWEfA8bNZvwn67V5SeQ0VAxSo0VFhDH6orZ88suLGZDSmHEfr+Oqxz7n04w9XkcTqbZUDBISmsbX4qnv9eDVH/ci3IwfvrSQ0f9JZ9sBbV4SKU3FICGlf/tEpt/bn18N7Mic9XsZMG42T32aSV6BNi+JnKRikJATHRHOHZe0Y+YvL+bSjo34x4wMBj02h9nrsnVynAg6XFWEzzL2MHbyKjbvO0pSg1oM7tqUQd2a0r1FvKYXlRpFcz6LnIHjJwqZvGwH01fsZG7mXk4UOprXq8Wgrk0YfF5Tzk+qp5KQoKdiEDlLh46e4OM1u5m+Yidz1u8lv7CIZvExDOzalCHnNeH8pPqEhakkJPioGETOgcPHT/DJmt1MXb6Lz9dnk19QRJO6MQzs2oTB3ZqS1kolIcFDxSByjh05foJZa/cwdflOPlvnK4lGdaIZ1LUJg7o15TvJDQhXSUg1pmIQCaCcvAJmrd3D9BU7+TRjD8dPFJFQO5qBXRszuFtTeiU3ICJcB/1J9aJiEKkiuXkFfJqxh+krdjFr7R6OnSikYVwUV3VtwuCuTendRiUh1YOKQcQDR/MLmJ2RzdQVO5m1dg9H8wtpEBfFVV0aM6hrU/q0bUikSkI8omIQ8djxE4V8lpHNtBU7+WTNbnLzC6kXG8mVKb7NTRe2TSAqQiUhVUfFIFKNHD9RyOfrfCUxc80ecvIKiK8VyRUpjRnSrSl926kkJPCqRTGYWRLwH3wzuTlggnPu8VJjLgHeBzb5F73jnHvoVO+rYpBglldQyJx1e5m2cicfr97NkeMF1ImJ4IqUxgzu2pT+HRKIjgj3OqbUQBUthogA5ygAfumcW+yf+3mRmX3snFtdatwc59zQAGcRqRaiI8IZkNKYASmNySso5MvMfUxdsZOPVu3incXbSagdxT9v6M4lHRt5HVVCVEDXXZ1zO51zi/33jwBrgOaB/EyRYBIdEc6lnRrxzxu6k/77K3jph98hoXY0o15ayMPT1pBfUOR1RAlBVbZR08ySgfOB+WU83cfMlpnZdDPrUlWZRKqTqIgwLu3YiPfu7MstvVvy7OcbueHZr9iyT3NGSNWqkmIws9rA/4B7nXOHSz29GGjlnOsOPAG8V857jDazdDNLz87ODmxgEQ/FRIbzl+924+mRPdiYncOQ8XOYsnyH17EkhAT8qCQziwSmADOcc+MqMH4zkOac21veGO18llCxdf9R7p60hCVbDjKiVxJ/HNqFWlHaMS1np6I7nwO6xmC+6xS/AKwprxTMrIl/HGbWy59pXyBziQSLpAaxvPWzPtx+SVsmLdzKsCfnkrHriNexpIYL9KakvsCtwGVmttR/G2xmt5nZbf4x1wMrzWwZMB642QXjyRUiARIZHsavB3biPz/qxYGj+Qx7ci6vz9+i2eYkYHSCm0gQ2XPkOL98axlz1u9lSLem/O3absTXivQ6lgSJarEpSUTOrUZ1Ynjlh7349cBOfLhqF0PGz2HJlgNex5IaRsUgEmTCwozbL2nLWz/rg3NwwzNf8ezsDRQVBd/av1RPKgaRINWzVX2m3dOfK1Ia8/D0tYx6eSF7c/K8jiU1gIpBJIjF14rk3yN78NdrujJ/4z4GPT6HuevLPdJbpEJUDCJBzswYeUEr3h/Tl/hakdz64nz+MWMtBYW6nIacHRWDSA3RqUldJo/py409k3jq0w3cNGEe2w7ochpy5lQMIjVIbFQEj1x/Ho/fnErGriMMfnwOH67c6XUsCTIqBpEaaHhqc6be3Y/khDhum7iYP7y3kuMnCr2OJUFCxSBSQ7VqGMfbt13IT/u35tV5WXz3qS/I3JPjdSwJAioGkRosKiKMB4ak8NKo77DnSB5XPzGX/6Zv1eU05JRUDCIh4NJOjZh+T39Sk+px/9vL+fmbS8nJK/A6llRTKgaRENG4bgwTf3IBv7iiA5OX7WDo+Dms2HbI61hSDakYREJIeJhx9+XtmTS6D3kFRVz79Be8MHeTNi1JCSoGkRDUq3UDpt3dn4s7NOLPU1bzk1fS2Z+b73UsqSZUDCIhqn5cFM99vydjr05hzvq9DHr8c+Zt1BxZomIQCWlmxqi+rXnnjguJjYrge8/N418fr6NQV2oNaQEvBjMbaGYZZpZpZr8p4/loM3vT//x8M0sOdCYRKalr83g+uKsf301tzuOfrGfEc/PYeeiY17HEI4Ge8zkceAoYBKQAI8wspdSwHwMHnHPtgH8BjwQyk4iUrXZ0BONuSuXRG7qzcvshhoyfy5z12V7HEg8Eeo2hF5DpnNvonMsHJgHDS40ZDrziv/82cLmZWYBziUg5ruvZgg/u6kdi7Wi+/+ICHp+5XpMAhZhAF0NzYGuxx9v8y8oc45wrAA4BDQOcS0ROoW1ibd6980KuSW3Ov2auY9TLC3XUUggJmp3PZjbazNLNLD07W6u3IoEWGxXBozd256/XdGXehn0MHT+HpVsPeh1LqkCgi2E7kFTscQv/sjLHmFkEEA9865g559wE51yacy4tMTExQHFFpLiTkwC9fXsfzIwbnvmS/3y1WSfE1XCBLoaFQHsza21mUcDNwORSYyYDP/Dfvx6Y5fR/nUi1cl6Leky9ux/92yfyx/dXcc+kpeTqWks1VkCLwb/PYAwwA1gDvOWcW2VmD5nZMP+wF4CGZpYJ/AL41iGtIuK9erFRPP/9NO6/qiNTlu9g+FNfkLnniNexJAAsGH85T0tLc+np6V7HEAlZX2Tu5e43lnDsRCF/v+48hnVv5nUkqQAzW+ScSzvduKDZ+Swi1UffdglMvbs/KU3rcvcbS3jw/ZXkFxR5HUvOERWDiJyVJvExvDG6Nz/p15pXvsrixme/YvtBnS1dE6gYROSsRYaH8fuhKTw9sgeZe3IYOn4Os9fpcPJgp2IQkUob1K0pk8f0pXHdGEa9tIBxuhBfUFMxiMg50SaxNu/e0Zdrzm/O+E/WM+qlBTpbOkipGETknKkVFc6jN3Tn4Wu7MX/TfoaMn8PiLQe8jiVnSMUgIueUmTGiV0veuf1CIsKNm579ipe/0PShwUTFICIB0bV5PFPG9OfiDomM/WA1d72xhBydLR0UVAwiEjDxsZFMuDWNXw3syLQVOxn+5FzW7dbZ0tWdikFEAioszLjjknZM/MkFHDp2guFPfsH7S0tfS1OqExWDiFSJC9v6zpbu1jyeeyYt5Q/vrSSvoNDrWFIGFYOIVJnGdWN47acXMPqiNrw6L4sbn/mKbQeOeh1LSlExiEiVigwP43eDO/PMLT3ZmJ3L0Cfm8mnGHq9jSTEqBhHxxMCuTZh8Vz+a1I3hRy8vZNxHGTpbuppQMYiIZ1onxPHenX25vkcLxs/KZNRLC9iXk+d1rJCnYhART8VEhvOPG7rzyHUnz5aey6IsnS3tpYAVg5n9w8zWmtlyM3vXzOqVM26zma0ws6Vmptl3RELUTd/xnS0dFRHGTc9+xYtzdba0VwK5xvAx0NU5dx6wDvjtKcZe6pxLrcjMQiJSc3VtHs8Hd/Xjko6NeGjKasa8voQjx094HSvkBKwYnHMf+ed8BpgHtAjUZ4lIzRFfK5Lnvt+T3wzqxPSVO7nrjSVac6hiVbWP4UfA9HKec8BHZrbIzEZXUR4RqcbMjNsubsvvBnfms4xsPlmjw1mrUqWKwcxmmtnKMm7Di415ACgAXivnbfo553oAg4A7zeyicj5rtJmlm1l6drZmiBIJBT+4MJl2jWrz0JTVHD+hs6SrSqWKwTk3wDnXtYzb+wBmNgoYCox05awLOue2+//cA7wL9Cpn3ATnXJpzLi0xMbEysUUkSESGh/Hg1Sls2X+UF+Zu8jpOyAjkUUkDgV8Bw5xzZZ7zbmZxZlbn5H3gSmBloDKJSPDp3z6Rq7o05slZmew8dMzrOCEhkPsYngTqAB/7D0V9BsDMmpnZNP+YxsBcM1sGLACmOuc+DGAmEQlCvx+SQpFzPDxtrddRQkJEoN7YOdeunOU7gMH++xuB7oHKICI1Q1KDWH52cVvGf7KeW3q3olfrBl5HqtF05rOIBIXbL25Ls/gYHpy8StdUCjAVg4gEhVpR4TwwJIU1Ow/z+oItXsep0VQMIhI0BndrQp82DXn0owwO5OZ7HafGUjGISNAwMx4clsKR4wWM+3id13FqLBWDiASVTk3qcmvvVrw2P4vVOw57HadGUjGISND5+YAOxNeKZOzkVbqOUgCoGEQk6MTHRnL/VZ1YsHk/Hyzf6XWcGkfFICJB6abvJNG1eV3+NnUNR/MLTv8CqTAVg4gEpfAwY+zVXdh1+DhPfZrpdZwaRcUgIkErLbkB15zfnOc+30TWvlyv49QYKgYRCWq/GdSJyHDjz1PWeB2lxlAxiEhQa1w3hrsub8/MNbv5LEMT+pwLKgYRCXo/7JtM64Q4HvpgNfkFRV7HCXoqBhEJetER4fxxaAob9+by8pea0KeyVAwiUiNc2qkRl3dqxOMz17Pn8HGv4wQ1FYOI1Bh/GJrCiULHIx9meB0lqAVyas+xZrbdP3vbUjMbXM64gWaWYWaZZvabQOURkZovOSGOH/dvzf8Wb2NR1gGv4wStQK8x/Ms5l+q/TSv9pJmFA08Bg4AUYISZpQQ4k4jUYGMubUfjutGMnbyKIk3oc1a83pTUC8h0zm10zuUDk4DhHmcSkSAWFx3B7wZ3ZsX2Q/x30Vav4wSlQBfDGDNbbmYvmln9Mp5vDhT/L7fNv0xE5KwN696MtFb1+b8PMzh07ITXcYJOpYrBzGaa2coybsOBp4G2QCqwE3i0kp812szSzSw9Ozu7Mm8lIjWcmTF2WBf2H83nsZma0OdMRVTmxc65ARUZZ2bPAVPKeGo7kFTscQv/srI+awIwASAtLU0bDkXklLo2j2dEr5b856ssRvRqSYfGdbyOFDQCeVRS02IPrwFWljFsIdDezFqbWRRwMzA5UJlEJLTcd2VHakdH8KcPNKHPmQjkPob/M7MVZrYcuBT4OYCZNTOzaQDOuQJgDDADWAO85ZxbFcBMIhJCGsRF8csrO/BF5j5mrNrldZygYcHYomlpaS49Pd3rGCISBAoKixj6xFyOHC9g5i8uplZUuNeRPGNmi5xzaacb5/XhqiIiARURHsbYYV3YfvAYz36+wes4QUHFICI1Xu82DRl6XlOe/mwD2w4c9TpOtadiEJGQ8LvBnTGDv03ThD6no2IQkZDQrF4t7rykHdNW7OKLzL1ex6nWVAwiEjJ+elEbkhrU4k8frOJEoSb0KY+KQURCRkxkOH8YksK63TlMnJfldZxqS8UgIiHlipTG9G+fwLiP17EvJ8/rONWSikFEQoqZ8eDVKRzLL+QfMzShT1lUDCIScto1qsOoC5N5M30ry7cd9DpOtaNiEJGQdM+A9jSM04Q+ZVExiEhIqhMTya8HdmTxloO8t7TMizqHLBWDiISs63q0oHtSPR6evpYjxzWhz0kqBhEJWWFhxp+GdSH7SB5Pzsr0Ok61oWIQkZCWmlSPG9Na8OIXm9iQneN1nGpBxSAiIe/+qzoRExHOQx+s1oQ+qBhEREisE809A9oze102n6zZ43UczwVyas83zWyp/7bZzJaWM26zf6a3pWam2XdExBM/uDCZdo1q89CU1Rw/Ueh1HE8FrBicczc551Kdc6nA/4B3TjH8Uv/Y084sJCISCJHhYTx4dQpb9h/lhbmbvI7jqYBvSjIzA24E3gj0Z4mIVEb/9olc1aUxT87KZOehY17H8UxV7GPoD+x2zq0v53kHfGRmi8xsdBXkEREp1++HpFDkHA9PW+t1FM9UqhjMbKaZrSzjNrzYsBGcem2hn3OuBzAIuNPMLirns0abWbqZpWdnZ1cmtohIuZIaxPKzi9syedkOFmza73UcT1ggD80yswhgO9DTObetAuPHAjnOuX+ealxaWppLT9d+ahEJjGP5hVz+6GfEx0Yx5a5+hIeZ15HOCTNbVJF9uYHelDQAWFteKZhZnJnVOXkfuBJYGeBMIiKnVCsqnAeGpLBm52FeX7DF6zhVLtDFcDOlNiOZWTMzm+Z/2BiYa2bLgAXAVOfchwHOJCJyWoO7NaFPm4Y8+lEGB3LzvY5TpQJaDM65Uc65Z0ot2+GcG+y/v9E5191/6+Kc+2sg84iIVJSZ8eCwFI4cL2Dcx+u8jlOldOaziEg5OjWpy629W/Ha/CxWbj/kdZwqo2IQETmFnw/oQIO4aO6etIScvAKv41QJFYOIyCnEx0by5PfOJ2vfUX799vKQuMieikFE5DR6t2nIr67qyNQVO3nxi81exwk4FYOISAWMvqgNV6Y05uFpa1i4uWaf+KZiEBGpADPjnzd2p0X9Wtz52mKyj+R5HSlgVAwiIhVUNyaSp2/pyeHjJ7jrjcUUFBZ5HSkgVAwiImegc9O6/O2abszbuJ9/flQzz29QMYiInKFre7Rg5AUteWb2Bmas2uV1nHNOxSAichb+eHUK57WI5763lrF5b67Xcc4pFYOIyFmIjgjn3yN7EB5u3DZxEcfya850oCoGEZGz1KJ+LI/dlErG7iM88N6KGnPym4pBRKQSLunYiHsub887i7fzxoKtXsc5J1QMIiKVdPdl7bm4QyJjJ69i+baDXsepNBWDiEglhYUZj92USmKdaG6fuDjo529QMYiInAP146L498geZB/J4943l1JUFLz7GypdDGZ2g5mtMrMiM0sr9dxvzSzTzDLM7KpyXt/azOb7x71pZlGVzSQi4oXuSfV4cFgKs9dl88SsTK/jnLVzscawErgW+Lz4QjNLwTe1ZxdgIPBvMwsv4/WPAP9yzrUDDgA/PgeZREQ88b1eLbm2R3Me+2Qds9dlex3nrFS6GJxza5xzGWU8NRyY5JzLc85tAjKBXsUHmJkBlwFv+xe9Any3splERLxiZvz1u93o2LgO90xawrYDR72OdMYCuY+hOVD82K1t/mXFNQQOOucKTjFGRCSo1IoK55lbelJY6LjztcXkFQTXyW8VKgYzm2lmK8u4DQ90wGIZRptZupmlZ2cH5+qZiISO5IQ4Hr2xO8u2HeKhD1Z7HeeMRFRkkHNuwFm893YgqdjjFv5lxe0D6plZhH+toawxJzNMACYApKWlBe/ufhEJGVd2acJtF7flmdkb6NGyPtf1bOF1pAoJ5KakycDNZhZtZq2B9sCC4gOc7/zxT4Hr/Yt+ALwfwEwiIlXqvis70LtNAx54bwVrdh72Ok6FnIvDVa8xs21AH2Cqmc0AcM6tAt4CVgMfAnc65wr9r5lmZs38b/Fr4Bdmlolvn8MLlc0kIlJdRISH8cSIHtSNieT2iYs4fPyE15FOy4Lxok9paWkuPT3d6xgiIhWWvnk/N0+Yx2WdGvHsrT3xHZRZtcxskXMu7XTjdOaziEgVSEtuwG8Hd+aj1buZ8PlGr+OckopBRKSK/KhvMkO6NeWRD9fy1YZ9Xscpl4pBRKSKmBmPXH8eyQlx3PXGEnYfPu51pDKpGEREqlDt6AievaUnR/MLGPP6Yk4UFnkd6VtUDCIiVax94zo8fG03Fm4+wCPT13od51tUDCIiHhie2pxRFybz/NxNTFux0+s4JagYREQ88rvBnTm/ZT3u/+8yNmTneB3nayoGERGPREWE8e+RPYiODOe2VxeRm1dw+hdVARWDiIiHmsbX4okR57MhO4ffvrOC6nDSsYpBRMRjfdsl8MsrOzJ52Q5enZfldRwVg4hIdXD7xW25vFMj/jxlNYu3HPA0i4pBRKQaCAszxt2YSpP4GO58bTH7cvK8y+LZJ4uISAnxsZE8PbIn+3LzuWfSUgqLvNnfoGIQEalGujaP5y/DuzI3cy+PzVznSQYVg4hINXPjd5K4KS2JJ2ZlMmvt7ir/fBWDiEg19KfhXejSrC73TlrK1v1Hq/SzK1UMZnaDma0ysyIzSyu2/AozW2RmK/x/XlbO68ea2XYzW+q/Da5MHhGRmiImMpynR/YE4LZeu95cAAAH+UlEQVSJizh+orDKPruyawwrgWuBz0st3wtc7Zzrhm8e51dP8R7/cs6l+m/TKplHRKTGaNkwlsduTmXVjsOMnbyqyj63UsXgnFvjnMsoY/kS59wO/8NVQC0zi67MZ4mIhKLLOjXmrsvaMWnhVt5auLVKPrMq9jFcByx2zpV3UO4YM1tuZi+aWf0qyCMiElTuHdCBfu0S+MP7K1m5/VDAP++0xWBmM81sZRm34RV4bRfgEeBn5Qx5GmgLpAI7gUdP8V6jzSzdzNKzs7NP99EiIjVGeJjx+M2p9GrdgJjIwP8+b+figk1m9hlwn3MuvdiyFsAs4IfOuS8q8B7JwBTnXNfTjU1LS3Pp6emnGyYiIsWY2SLnXNrpxgWkesysHjAV+M2pSsHMmhZ7eA2+ndkiIuKhyh6ueo2ZbQP6AFPNbIb/qTFAO+CPxQ5FbeR/zfPFDm39P/8hrcuBS4GfVyaPiIhU3jnZlFTVtClJROTMebopSUREgpeKQURESlAxiIhICSoGEREpQcUgIiIlBOVRSWaWDZztjNkJ+C7yJz76Pr6h76IkfR8l1YTvo5VzLvF0g4KyGCrDzNIrcrhWqND38Q19FyXp+ygplL4PbUoSEZESVAwiIlJCKBbDBK8DVDP6Pr6h76IkfR8lhcz3EXL7GERE5NRCcY1BREROIaSKwcwGmlmGmWWa2W+8zuMVM0sys0/NbLWZrTKze7zOVB2YWbiZLTGzKV5n8ZqZ1TOzt81srZmtMbM+Xmfyipn93P/vZKWZvWFmMV5nCrSQKQYzCweeAgYBKcAIM0vxNpVnCoBfOudSgN7AnSH8XRR3D7DG6xDVxOPAh865TkB3QvR7MbPmwN1Amn8SsXDgZm9TBV7IFAPQC8h0zm10zuUDk4DTTk9aEznndjrnFvvvH8H3j765t6m85Z9xcAjwvNdZvGZm8cBFwAsAzrl859xBb1N5KgKoZWYRQCyww+M8ARdKxdAc2Frs8TZC/IchfD2l6vnAfG+TeO4x4FdAkddBqoHWQDbwkn/T2vNmFud1KC8457YD/wS24JuX/pBz7iNvUwVeKBWDlGJmtYH/Afc65w57nccrZjYU2OOcW+R1lmoiAugBPO2cOx/IBUJyn5yZ1ce3ZaE10AyIM7NbvE0VeKFUDNuBpGKPW/iXhSQzi8RXCq85597xOo/H+gLDzGwzvk2Ml5nZRG8jeWobsM05d3It8m18RRGKBgCbnHPZzrkTwDvAhR5nCrhQKoaFQHsza21mUfh2IE32OJMnzMzwbT9e45wb53Uerznnfuuca+GcS8b3/8Us51yN/62wPM65XcBWM+voX3Q5sNrDSF7aAvQ2s1j/v5vLCYEd8RFeB6gqzrkCMxsDzMB3ZMGLzrlVHsfySl/gVmCFmS31L/udc26ah5mkerkLeM3/S9RG4Ice5/GEc26+mb0NLMZ3NN8SQuAMaJ35LCIiJYTSpiQREakAFYOIiJSgYhARkRJUDCIiUoKKQURESlAxSLVlZjleZzgdM9tsZgn++19W4n1GmVmzc5dM5OypGERK8V8s7Yw55ypzRuwofJdcOKfO9u8ioU3FIEHFzJLNbJaZLTezT8yspX95WzObZ2YrzOwv5a1tmNkf/HNyzPVfW/8+//LPzOwxM0sH7jGzq81svv8icjPNrLF/XEMz+8h/ff7nASv23jnF7t9vZgv9Of9ULPsaM3vO//qPzKyWmV0PpOE7oWypmdUqlfkzM3vEzBaY2Toz6+9fHmNmL/n/zkvM7FL/8lFmNtnMZgGfmNklZjbbzN43s41m9nczG+l/vxVm1vac/QeSGkHFIMHmCeAV59x5wGvAeP/yx4HHnXPd8F3r51vM7DvAdfjmFxiE74dxcVHOuTTn3KPAXKC3/yJyk/BdeRXgQWCuc64L8C7QsozPuRJoj+9S76lATzO7yP90e+Ap/+sPAtc5594G0oGRzrlU59yxMuJHOOd6Aff6MwDcCTj/33kE8EqxSWR6ANc75y72P+4O3AZ0xnfWewf/+z2P7yxnka+pGCTY9AFe999/FehXbPl//fdfL/0iv77A+8654/55KD4o9fybxe63AGaY2QrgfqCLf/lFwEQA59xU4EAZn3Ol/7YE36UUOuErBPBdkO3kZUgWAcnlZC3t5IUOi7+mX7Esa4EsoIP/uY+dc/uLvX6hfx6OPGADcPLS0SvOIIOECBWDyDdyi91/AnjS/9v4z4Azmc7RgIf9v/2nOufaOede8D+XV2xcIRW/XtnJ11X0NbmlHhf/3KJij4vOIIOECBWDBJsv+WZqxZHAHP/9efg2E0H5Uy9+AVzt3zZfGxh6is+J55vLsv+g2PLPge8BmNkgoH4Zr50B/Mj/GZhZczNrdIrPAjgC1DnNmNLm4PsOMLMO+DZrZZzhe4h8i35TkOos1syK7y8Yh297+Etmdj++WcZOXvXzXmCimT0AfAgcKv1mzrmFZjYZWA7sxrcZ5Vvj/MYC/zWzA8AsfBO1APwJeMPMVuErqS1lfM5HZtYZ+Mp3pWZygFvw/bZfnpeBZ8zsGNCnnP0Mpf0beNq/uasAGOWcy/N/pshZ09VVpUYws1jgmHPOmdnNwAjn3Lfm9Daz2s65HP/4z4HRJ+e/FhEfrTFITdETeNI/mcpB4EfljJtgZin49hm8olIQ+TatMYiISAna+SwiIiWoGEREpAQVg4iIlKBiEBGRElQMIiJSgopBRERK+H9Yay+A0DgmYAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print('\\nOptimizing with custom printing and logging:')\n",
    "custom_objective.reset()\n",
    "custom_objective.set_print_every(1)\n",
    "custom_objective.set_log_every(1)\n",
    "opt_result = Optimize(custom_objective)\n",
    "print(opt_result.message)\n",
    "\n",
    "plt.plot(\n",
    "    [ entry[0] for entry in custom_objective.optimization_log],\n",
    "    np.log([ entry[1] for entry in custom_objective.optimization_log]))\n",
    "plt.xlabel('Iteration')\n",
    "plt.xlabel('Log gradient norm')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}