src/Hopfield networks tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hopfield neural networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What is Hopfield Neural network?**\n",
    "\n",
    "This is a network of interconnected nodes (neurons), each governed by dynamic equations of specific form. In the simplest formulation (discrete, and symmetric connectivity matrix), the neuron can be only in two states: +1 (up), and -1 (down). The neuron affects all the connected to it neighboring neurons by synapses: positive connections force the two connected neurons be in the same state, and negative connections tend to make neurons be in the opposite states."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='../imgs_for_jupyter/HN.png' width=\"240\" height=\"240\" align=\"center\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Equations for discrete Hopfiel Neural network of size $N$ (vector form)**:\n",
    "\n",
    "\n",
    "$$h[n+1] =  Ws[n] + b $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$s[n+1] = sgn(h[n+1])$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "where\n",
    "$\\textbf{n}$ represents a number of a step during the evolution of the system\n",
    "\n",
    "$\\textbf{h}$ - is the column vector hidden state (could be regarded as a membrane potential of a neuron, or $\\textbf{local field}$),\n",
    "\n",
    "$\\textbf{s}$ - is the column vector of states of neurons, (The variable ${ (\\boldsymbol{s_{i}}+1)/2 }$ could be considered as firing rate of $i_{th}$ neuron)\n",
    "\n",
    "$\\textbf{W}$ - is the connectivity matrix $N \\times N$, where the synaptic weights are stored, \n",
    "\n",
    "$\\textbf{b}$ - is the bias (could be considered as a constant current injected into a neuron)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This network is very similar to dynamics of the atomic spins interacting with each other in solid states.\n",
    "Thus one can define an energy function:\n",
    "\n",
    "For two neurons, the energy of their interaction is : $$H_{i,j} = -w_{ij}s_i s_j$$\n",
    "\n",
    "For the whole system of $N$ neurons:\n",
    "\n",
    "$$ H = - s^{T} W s - (b,s)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This energy function has the local extrema - the fixed points of the dynamics. Once the network arrives at this states, it no longer escapes it (if there is no noise). If we set $b = 0$ The positions of these local minima are encoded in the weight matrix $W$. This makes the system ideal for storage of binary vectors (patterns). \n",
    "\n",
    "Once the system has learned (the weights has been adjusted accordingly) to store the patterns, one can initialise the system in one of these patterns corrupted with noise, and if the noise level is sufficiently small, the system will relax to the state corresponding to the closest pattern it has memorised."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hebbian learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The simplest rule to learn these patterns by network is: 'The neurons which fire together - wire together' - famous **Hebb's rule**. Lets call the $\\textbf{p}$ patterns to memorise by $\\boldsymbol{\\sigma^{\\nu}}$, where $\\nu \\in \\{1,.,p\\}$. \n",
    "\n",
    "In the context of Hopfield Neural networks the rule will be represented as:\n",
    "$$w_{i,j} = \\frac{1}{N}\\sum_{\\nu}\\boldsymbol{\\sigma^{\\nu}_{i}}\\boldsymbol{\\sigma^{\\nu}_{j}}$$\n",
    "\n",
    "or in vectorised form (if we set $Z = \\{\\boldsymbol{\\sigma^1} ,\\boldsymbol{\\sigma^2} \\dots \\boldsymbol{\\sigma^p}\\}$ - a $N \\times p$ matrix consisting of collumn vectors of patterns stacked together horizontally):\n",
    "\n",
    "$$W = \\frac{1}{N}ZZ^T$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's call the $\\boldsymbol{m^{\\nu}} = \\frac{1}{N}(\\sigma^{\\nu},s)$ - an overlap of the current state $s$ with the pattern $\\sigma^{\\nu}$. If we consider an overlaps between two patterns - $\\sigma^{\\nu}$ and $\\sigma^{\\mu}$ - we denote their overlap as $M^{\\mu\\nu}$, where $M$ is the matrix of overlaps of patterns (**Important note**: $M = \\frac{1}{N}Z^T Z$)\n",
    "\n",
    "Consider $s[n]$ to be equal one of the patterns, say $\\sigma^{\\nu}$. Then the state on the next time step is:\n",
    "\n",
    "$$s[n+1] = sgn\\Big(\\frac{1}{N}ZZ^T s[n]\\Big) = sgn\\Big(\\frac{1}{N}ZZ^T \\sigma^{\\nu}\\Big) = sgn\\Big(ZM[:,\\nu]\\Big) = sgn\\Big(\\sum_{\\mu} M^{\\mu\\nu}\\sigma^{\\mu}\\Big) = sgn\\Big(\\sigma^{\\nu} + \\sum_{\\mu \\neq \\nu} M^{\\mu\\nu}\\sigma^{\\mu}\\Big)$$\n",
    "\n",
    "where $M[:,\\nu]$ is a size $p$ column vector of overlaps of patterns in $Z$ with $\\sigma^{\\nu}$. If the overlaps of $\\sigma^{\\nu}$ and other $\\sigma^{\\mu}$ are sufficiently small (which is usually the case for random binary vectors and large $N$) then $s[n+1]$ is exactly equal to $\\sigma^{\\mu}$, which means that the state is a fixed point of the dynamics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The asymptotic storage capacity of Hopfield Network with hebbian rule is $O\\big(\\frac{N}{2 \\ln N}\\big)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEYCAYAAABGJWFlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEJBJREFUeJzt3XuQZGV9xvHvA6vcFnZFsXS5RhGisXT9wyxWJUoqEiSExHgjsZKIhqQwMUml0MJLGSkFY9REtEjU8kokaxSNBkkpWTWoKFZKC4zxglfMmnWBlV0BgcS4b/44Z6R3/M1Mz0zvdPfM91PVtd3n9r7nvOec57x9es6mtYYkSbMdMO4KSJImkwEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKk0kQERJJrkpw7a1hLcuIK1uHBSa5MsqMv+4SVKnvSTUj7nJnk2iR7kuxM8tYkh69U+ZNsQtrnl5J8sW+f7yf5QJKjV6r8STYJ7TOr7LcPW/5EBMSoJVm3hNn2Ah8Bnjri6miWJbbPBuAiYBPwcOBo4DWjrJc6S2yfLwOnt9Y20rXR14E3jrRiApbcPjPz/gLw0KFnaK2N5AXcBLyIbkfZDbwDOLgfdz/gKuDWftxVwDH9uIuBHwP3AHcClwKfBBrww37Y2f20vwbcAOwBPgM8alb5FwD/AfwPsK4f9vx+2A+A98zUaZ71WNeXfcKots0kvFZL+wws7ynAF8e9XW2fcl0OAv4S+PK4t6vts886rAOuBx7Vl3/igus94g34n8CxwJHAp4GL+nH3p7syPxQ4HLgC+ODAvNcA585a3j4rADwGuAXYAhwIPKsv86CB8m/oyz9kYNi/013RHAl8BThvgfVYzQEx9e0zUN4lwD+Oe7vaPvuUeRzdyW0v8CPgnHFvV9tnnzJfALy+Kn/OeUa8Ac8b+PyrwDfnmHYzsHuRG/CNwCtmTXMj8ISB8p9T1Ol3Bj6/GnjTAuuxmgNi6tunn+40uiu1k8a9XW2fsn5H0l3tnjLu7Wr7/GTcscA3gA1V+XO9Rn0PYvvA++/QJRtJDk3y5iTfSXI7XRdrY5IDF7Hs44Hz+5tge5LsoVvpTXOUP2PnwPu7gPWLKHO1mfr2SXIKsBV4Wmvta4uo3zSY+vYBaK3dBlwG/PNyvi+fQNPcPpcAL2+t/WARdRp5QBw78P44YEf//nzgZGBLa+0I4PH98PT/tiGWvR24uLW2ceB1aGvt3QPTDLOctWyq2yfJY4Ar6a6kPracZU2oqW6fWdYBDwSOGOEyx22a2+eXgdf0vwCcCZXrkjxzvplGHRB/nOSYJEcCL6G7aQLd93J3A3v6cS+bNd/NwEMWGPYW4LwkW9I5rP/p48h+6pjkYLobbAAH9Z9Xk6ltnySPpPuV2Z+01j40imVOoGlun6ckOTnJAUmOAv4GuL7vTawWU9s+wEnAo+m+/trcDzsL+MB8M406ILYC/wp8C/gm3c8SoeveHALsAj5Ld6APej3wtCS7k7yhH3YhcFnf3XpGa+1zwB/Q/QpgN933aeeMuP530/2qAOCr/efVZJrb53zgKOBtSe7sX18a4fInwTS3z9F9ve4Avkh3o/o3R7j8STC17dNau6W1tnPm1Q/e1Vqb9xyX/obFsiW5ie5GzEdHskCNlO0z2WyfybZW22dV/qGcJGn5DAhJUmlkXzFJklYXexCSpNJ++SOW0w54+tR3S7btvSILTzWdbJ/Jtnfnw6a+fQ540Ndtnwk2bPvYg5AklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklSbuvwO8escNPzXs9E2biylHM58Wx/bZ/0a9Xaptv7/L3LZ3pIubSpO8fw/bPvYgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEmliftLaklaDYb5C/ZJZw9CklQyICRJJQNCklSauHsQS30C4iQ/OXE1sX2ktcMehCSpZEBIkkoGhCSpZEBIkkordpN6Uv7LyUmpx7Qa5R//uN21mk3y/u1/OSpJWhYDQpJUMiAkSSUDQpJUWrGb1NUNm/19w9gb0stj+0hrmz0ISVLJgJAklQwISVLJgJAklcb6uO9hb1yPcvmqjeOG8Th+uCCtFP/LUUnSqmVASJJKBoQkqWRASJJKq+b/pNbyTMp2n5R6SLIHIUmagwEhSSoZEJKkkgEhSSqltTbuOkiSJpA9CElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMiCEl+XCSZ80z/k1JXjrksq5Jcu7oaidNliQvTvLWMZTbkpy40uWuVms6IJLclOSJw0zbWjujtXZZP985Sa6dNf681torRlCnjUnenmRnkjuSfC3JCwfGL+oAMIw0W7/f353kzn4/e2eS9UPOe2qS7y40XWvtla21idrv+mPhnn69dyX5pyQPHmK+E/rjbt3AsJ86B6xGazogJtTrgPXAw4ENwK8D3xhrjbQandVaWw9sBh4DvGhUCx48kU6g5/XrfRKwke54W3ETvo1+woDozVwRJHltkt1Jvp3kjIHx1yQ5N8nDgTcBj+uvRPb049+Z5KL+/f2SXJXk1n5ZVyU5ZsiqPBbY2lrb3Vrb21r7amvtff1yP9lP84W+7LPnKyvJxcAvApf201/aD//ZJNuS3JbkxiTPGMEm1BRqre0ErqYLCgCSHNQfB/+V5Ob+69NDkhwGfBjY1O9PdybZlOTCJO9LcnmS24Fz+mGXDyzzlCSfSbInyReSnNoPPzvJ5wbrlOTPk1w5X10Gpn1Bku8l2ZHkOYtY79uA9wOP7JdzZpLrk9yeZHuSCwcmnznu9vTr/Djqc8CcdZ3peSW5IMlO4B0Dw85Pcku/Hs8edh1WggGxry3AjcADgFcDb0uSwQlaa18BzgOua62tb61tLJZzAPAO4HjgOOBu4NIh6/BZ4OIkz07ysFllP75/++i+7PfMV1Zr7SXAp+ivmlprz+sP8m3AVuCBwG8Bf5fkEUPWT6tIfzFxBvv2Ul9Fd4W9GTgROBr4i9baD/tpd/T70/rW2o5+nt8A3kd3Vf4Ps8o4GvgX4CLgSOD5wPuTHAV8CDh51r7+TLr9c8669Mt9Ur+s04CHAUN9XdzP+wDgqcD1/aAfAr/X1/9M4LlJntyPmznuNvbrfB31OWDOuvYe1K//8cAfDgzb0E/7+8DfJrnfsOux37XW1uwLuAl4Yv/+HOAbA+MOBRrwoP7zNcC5A9NeO2tZ7wQumqOczcDugc8/WVYx7SHAi4HPAz+iO3DPGBjfgBPnWad5ywLOBj41a543Ay8bd3v4WplXv9/fCdzR708fozv5AYTuZPnQgekfB3y7f38q8N1Zy7sQ+GQx7PL+/QXAu2aNvxp4Vv/+croAgu5Ef0d//C1Ul7cDrxoYd9J8x0d/LNwF7AH+my7Ijppj2kuA1/XvT+iXu25g/D7ngCG32/8CBw+MP5Xugm5wubcAp4x7H5l5TcX3YCto58yb1tpdfedhqJt3g5IcSvfd5pOAmauBw5Mc2Fr78XzzttbuBl4JvDLJEcALgSuSHNe6bvFyyzoe2DLTLe6tA941/BpqFXhya+2jSZ5Ad7X+ALoT51F0J+fPD3SeAxy4wPK2zzPueODpSc4aGHYf4N/691uBvwZeTtd7+GB//D1wgbpsoruQmvGdBeoI8KettZ/6dVWSLXQ9gEcC9wUOAq4YYnkzhtlut7bW7pk13/dba/838PkulnDO2V/8imlp2gLjzwdOBra01o7g3i5q5p6lKKS12+nC4jDgZ5ZY1uy6bgc+0VrbOPBa31p77mLqptWhtfYJut7va/tBu+iuan9uYP/Y0LobuzD3vj/fMbGdrgcxuM8d1lp7VT9+G3BUks3Ab3Pv10sL1eV7wLED5Rw33FqXtgJXAse21jbQ3WOY6xiqhi1U17mWM9EMiKW5GTgmyX3nGH843c6yJ8mRwMuGXXCSlyZ5bJL7JjkY+DO6K7sbB8p+yCLKmj39VcBJSX43yX3612PT3XzX2nQJcFqSR7fW9gJvAV7XX8GT5Ogkp/fT3gzcP8mGRSz/cuCsJKcnOTDJwf0N2mMAWms/ortafw3dd/Tb+uEL1eW9dDfEH9H3pIc+zgqHA7e11u5J8vN0PZkZtwJ72fc42uccMERdp5IBsTQfB74E7Eyyqxh/Cd29hF10N50/sohlN7qbzruAHXQ34M5srd3Zj78QuKz/Ncgzhijr9cDT0v3C6Q2ttTuAX6G7Ob2D7mu1v6LrUmsNaq3dCvw9995QvYDu3tdn+18lfZSul0pr7avAu4Fv9fvgpiGWv53uJvaL6U6224EXsO/5ZyvdTeYrZn3lMl9dPky3/3+8n+bji175e/0R8PIkd9Bth/cO1P8u4GLg0/06n0J9DpizrtMq/Y0RSZL2YQ9CklQyICRJJQNCklQyICRJpRX7Q7nTDnj61NwNv3rHDSta3umbNi88UWGp9VxqeeOwbe8Vi/rbkdVsmo4hTY7lHEP2ICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklRasYf1TZNpeZjdtNRT0nSyByFJKhkQkqSSASFJKhkQkqSSASFJKhkQkqSSASFJKhkQkqSSASFJKhkQkqSSASFJKhkQkqSSASFJKvk018LVO25Y0nw+XVXqTMsxNC31HBd7EJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSqt2sd9L/UxvuPgI4clTSJ7EJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKk0qp9mutynnS60k+CXWpdfQqspP3JHoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqTTxT3MdxxNLp+Vpp9NST60907JvTks9x8UehCSpZEBIkkoGhCSpZEBIkkoGhCSpZEBIkkoGhCSpZEBIkkoGhCSpZEBIkkoGhCSpZEBIkkoGhCSpNPFPc10LxvHE2qWaprpqfJa6n0yL5ezP03QM2YOQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUm/mmuS32C4XKeJrnST01c6fKmadtofNxP5rZWto09CElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUm/nHfSzWOR+ou5xHAK2maHjes8XE/mdta2Tb2ICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJpVX7NNdxWCtPeJS0NtiDkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSV0lobdx0kSRPIHoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqfT/857PzNBYhakAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from visualisation import run_visualisation\n",
    "learning_options = {'sc' : True, 'incremental' : True}\n",
    "retrieval_options = {'time_of_retrieval' : 100, 'sync' : True}\n",
    "options = {'rule' : 'Hebb', 'learning_options' : learning_options, 'retrieval_options' : retrieval_options}\n",
    "run_visualisation(options=options, flips = 10, pattern_num = 1, random_patterns = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pseudoinverse rule "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But Hebbian rule is quite limited in terms of memory capcacity. A better option is **pseudoinverse** rule:\n",
    "\n",
    "$$W = ZZ^+$$\n",
    "\n",
    "where $^+$ denotes Moore-Penrouse inversion: $Z^+ = (Z^TZ)^{-1}Z^T$\n",
    "\n",
    "Consider, once again, $s[n] = \\sigma^{\\nu}$. Then the state on the next time step is:\n",
    "\n",
    "$$s[n+1] = sgn\\Big(ZZ^+ s[n]\\Big) = sgn\\Big(Z(Z^TZ)^{-1}Z^T \\sigma^{\\nu}\\Big) = sgn\\Big(ZM^{-1}M[:,\\nu]\\Big) = sgn(\\sigma^{\\nu})$$\n",
    "\n",
    "So, if there is no noise in the system, and the network is already in state which is represented by one of the patterns, it stays there. And it is much more stable than Hebbian rule!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The asymptotic storage capacity of Hopfield Network with pseudoinverse learning rule is $O(N)$. But, it is neither local, nor incremental, so it is not biologically plausible, as the network can't learn patterns one by one (non-incremental), and for the weight update, the synapse requires information which is not available at the spot (non-local) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEYCAYAAABGJWFlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEKtJREFUeJzt3XuQZGddxvHvs1nIbZNdAqHC5irkIkjB8gduqFKIJTHEGEUgRCmVgNEKilpWoMKlkBQkiIAmUFGguEbiIgQEQyyICxjulAWVIHIJ1+DiskmW7JKEJIrs6x/nDOkdfjPTM9s7093z/VR1bfe5ve857znnOW+fnrNprSFJ0mxrVroCkqTxZEBIkkoGhCSpZEBIkkoGhCSpZEBIkkoGhCSpNBYBkeT6JOfPGtaSnLiMdXhIkmuSbO/LPmG5yh53Y9I+ZyX5ZJLdSXYkeXOSw5ar/HE2Ju3zS0m+2LfP95O8L8nRy1X+OBuH9plV9luHLX8sAmLUkqxdwmx7gA8BTx1xdTTLEttnPXAJsBF4OHA08OpR1kudJbbPl4EzWmsb6Nro68DrR1oxAUtun5l5fwF42NAztNZG8gJuBl5It6PsAt4GHNSPewBwLXBbP+5a4Jh+3KXAj4F7gbuAK4CPAw34YT/s3H7aXwNuBHYDnwYeNav8i4D/AP4HWNsPe14/7AfAu2bqNM96rO3LPmFU22YcXtPSPgPLewrwxZXerrZPuS4HAn8JfHmlt6vts9c6rAVuAB7Vl3/igus94g34n8CxwBHAp4BL+nEPpLsyPwQ4DLgaeP/AvNcD589a3l4rADwGuBXYDBwAPLMv88CB8m/syz94YNi/013RHAF8BbhggfWY5oCY+PYZKO9y4B9XervaPnuVeRzdyW0P8CPgvJXerrbPXmU+H3htVf6c84x4A14w8PlXgW/OMe0mYNciN+DrgZfPmuYm4AkD5T+7qNPvDHx+FfCGBdZjmgNi4tunn+50uiu1k1d6u9o+Zf2OoLvaPXWlt6vt85NxxwLfANZX5c/1GvU9iG0D779Dl2wkOSTJG5N8J8kddF2sDUkOWMSyjwcu7G+C7U6ym26lN85R/owdA+/vBtYtosxpM/Htk+RUYAvwtNba1xZRv0kw8e0D0Fq7HbgS+Od9+b58DE1y+1wOvKy19oNF1GnkAXHswPvjgO39+wuBU4DNrbXDgcf3w9P/24ZY9jbg0tbahoHXIa21dw5MM8xyVrOJbp8kjwGuobuS+si+LGtMTXT7zLIWeDBw+AiXudImuX1+GXh1/wvAmVD5TJJnzDfTqAPij5Mck+QI4MV0N02g+17uHmB3P+6ls+a7BXjoAsPeBFyQZHM6h/Y/fRzZTx2THER3gw3gwP7zNJnY9knySLpfmf1Ja+0Do1jmGJrk9nlKklOSrElyJPA3wA19b2JaTGz7ACcDj6b7+mtTP+xs4H3zzTTqgNgC/CvwLeCbdD9LhK57czCwE/gs3YE+6LXA05LsSvK6ftjFwJV9d+vprbXPAX9A9yuAXXTfp5034vrfQ/erAoCv9p+nySS3z4XAkcBbktzVv740wuWPg0lun6P7et0JfJHuRvVvjnD542Bi26e1dmtrbcfMqx+8s7U27zku/Q2LfZbkZrobMR8eyQI1UrbPeLN9xttqbZ+p/EM5SdK+MyAkSaWRfcUkSZou9iAkSaX98kcsp685Z+K7JVv3XJ2Fp5pMts9427PjpIlvnzVHfd32GWPDto89CElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSaez+O8Drtt/4U8PO2LipmHI082lxbJ/9b9Tbpdr2+7vMrXtGuriJNM7797DtYw9CklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQau7+klqRpMMxfsI87exCSpJIBIUkqGRCSpNLY3YNY6hMQx/nJidPE9pFWD3sQkqSSASFJKhkQkqSSASFJKi3bTepx+S8nx6Uek2qUf/zjdtc0G+f92/9yVJK0TwwISVLJgJAklQwISVJp2W5SVzds9vcNY29I7xvbR1rd7EFIkkoGhCSpZEBIkkoGhCSptKKP+x72xvUol6/aStwwXokfLkjLxf9yVJI0tQwISVLJgJAklQwISVJpav5Pau2bcdnu41IPSfYgJElzMCAkSSUDQpJUMiAkSaW01la6DpKkMWQPQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDYkhJPpjkmfOMf0OSlwy5rOuTnD+62knjJcmLkrx5BcptSU5c7nKn1aoOiCQ3J3niMNO21s5srV3Zz3dekk/OGn9Ba+3lI6jThiRvTbIjyZ1JvpbkBQPjF3UAGEaard/v70lyV7+fvT3JuiHnPS3JdxearrX2itbaWO13/bFwb7/eO5P8U5KHDDHfCf1xt3Zg2E+dA6bRqg6IMXUZsA54OLAe+HXgGytaI02js1tr64BNwGOAF45qwYMn0jH03H69TwY20B1vy27Mt9FPGBC9mSuCJK9JsivJt5OcOTD++iTnJ3k48Abgcf2VyO5+/NuTXNK/f0CSa5Pc1i/r2iTHDFmVxwJbWmu7Wmt7Wmtfba29p1/ux/tpvtCXfe58ZSW5FPhF4Ip++iv64T+bZGuS25PclOTpI9iEmkCttR3AdXRBAUCSA/vj4L+S3NJ/fXpwkkOBDwIb+/3priQbk1yc5D1JrkpyB3BeP+yqgWWemuTTSXYn+UKS0/rh5yb53GCdkvx5kmvmq8vAtM9P8r0k25M8exHrfTvwXuCR/XLOSnJDkjuSbEty8cDkM8fd7n6dH0d9DpizrjM9ryQXJdkBvG1g2IVJbu3X41nDrsNyMCD2thm4CXgQ8CrgLUkyOEFr7SvABcBnWmvrWmsbiuWsAd4GHA8cB9wDXDFkHT4LXJrkWUlOmlX24/u3j+7Lftd8ZbXWXgx8gv6qqbX23P4g3wpsAR4M/Bbwd0keMWT9NEX6i4kz2buX+kq6K+xNwInA0cBftNZ+2E+7vd+f1rXWtvfz/AbwHrqr8n+YVcbRwL8AlwBHAM8D3pvkSOADwCmz9vVn0O2fc9alX+6T+mWdDpwEDPV1cT/vg4CnAjf0g34I/F5f/7OA5yR5cj9u5rjb0K/zZ6jPAXPWtXdUv/7HA384MGx9P+3vA3+b5AHDrsd+11pbtS/gZuCJ/fvzgG8MjDsEaMBR/efrgfMHpv3krGW9HbhkjnI2AbsGPv9kWcW0BwMvAj4P/IjuwD1zYHwDTpxnneYtCzgX+MSsed4IvHSl28PX8rz6/f4u4M5+f/oI3ckPIHQny4cNTP844Nv9+9OA785a3sXAx4thV/XvLwLeMWv8dcAz+/dX0QUQdCf6O/vjb6G6vBV45cC4k+c7Pvpj4W5gN/DfdEF25BzTXg5c1r8/oV/u2oHxe50Dhtxu/wscNDD+NLoLusHl3gqcutL7yMxrIr4HW0Y7Zt601u7uOw9D3bwblOQQuu82nwTMXA0cluSA1tqP55u3tXYP8ArgFUkOB14AXJ3kuNZ1i/e1rOOBzTPd4t5a4B3Dr6GmwJNbax9O8gS6q/UH0Z04j6Q7OX9+oPMc4IAFlrdtnnHHA+ckOXtg2P2Af+vfbwH+GngZXe/h/f3x9+AF6rKR7kJqxncWqCPAn7bWfurXVUk20/UAHgncHzgQuHqI5c0YZrvd1lq7d9Z832+t/d/A57tZwjlnf/ErpqVpC4y/EDgF2NxaO5z7uqiZe5aikNbuoAuLQ4GfWWJZs+u6DfhYa23DwGtda+05i6mbpkNr7WN0vd/X9IN20l3V/tzA/rG+dTd2Ye59f75jYhtdD2Jwnzu0tfbKfvxW4Mgkm4Df5r6vlxaqy/eAYwfKOW64tS5tAa4Bjm2trae7xzDXMVQNW6iucy1nrBkQS3MLcEyS+88x/jC6nWV3kiOAlw674CQvSfLYJPdPchDwZ3RXdjcNlP3QRZQ1e/prgZOT/G6S+/Wvx6a7+a7V6XLg9CSPbq3tAd4EXNZfwZPk6CRn9NPeAjwwyfpFLP8q4OwkZyQ5IMlB/Q3aYwBaaz+iu1p/Nd139Fv74QvV5d10N8Qf0fekhz7OCocBt7fW7k3y83Q9mRm3AXvY+zja6xwwRF0nkgGxNB8FvgTsSLKzGH853b2EnXQ3nT+0iGU3upvOO4HtdDfgzmqt3dWPvxi4sv81yNOHKOu1wNPS/cLpda21O4Ffobs5vZ3ua7W/outSaxVqrd0G/D333VC9iO7e12f7XyV9mK6XSmvtq8A7gW/1++DGIZa/je4m9ovoTrbbgOez9/lnC91N5qtnfeUyX10+SLf/f7Sf5qOLXvn7/BHwsiR30m2Hdw/U/27gUuBT/TqfSn0OmLOukyr9jRFJkvZiD0KSVDIgJEklA0KSVDIgJEmlZftDudPXnOPd8BG7bvuNS5rvjI2bFp5oxJZa1zVHfX1RfzsyzTyGtBRb91y95GPIHoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKy/awvuV+sNxSy9uXMpfbpNQTll7XrXtGXBFJQ7MHIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqLdvTXJf7yaP7Ut5yP3l2qSalnlp9JmXfnJR6rhR7EJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSot2+O+J8lqeZSvJM3HHoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqTS1T3O9bvuNy17mcj8F1qfOStqf7EFIkkoGhCSpZEBIkkoGhCSpZEBIkkoGhCSpZEBIkkoGhCSpZEBIkkoGhCSpZEBIkkoGhCSpZEBIkkpprS1LQXt2nLSkgnxi6eq2dc/VWek6jIvT15yzPAerpsq+HEP2ICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJpbXLVZBPZZ0O122/cUnz2f4rxzYbL5PUHvYgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEmlZXua63Jb6hMTYfqfYum2WV2W2mbuJ3NbLdvGHoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKBoQkqWRASJJKU/u47315pO6+PMp3EkzS44a1cjyG5rZajiF7EJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKk0tQ+zXVfnia5Wp7UKO0vHkPTwR6EJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKmU1tpK10GSNIbsQUiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSv8PRUnb7oFqrIIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from visualisation import run_visualisation\n",
    "learning_options = {}\n",
    "retrieval_options = {'time_of_retrieval' : 100, 'sync' : True}\n",
    "options = {'rule' : 'Pseudoinverse', 'learning_options' : learning_options, 'retrieval_options' : retrieval_options}\n",
    "run_visualisation(options=options, flips = 10, pattern_num = 1, random_patterns = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Storkey rule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is yet another rule, which performs pretty well, and at the same time is biologically plausible (it's both incremental and local)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$w_{ij} = \\sum_{\\mu}(\\sigma^{\\mu}_i - f_{ij})(\\sigma^{\\mu}_j - f_{ji})$$\n",
    "\n",
    "where $f_{ij}$ is a local field on the neuron $i$ except the contribution of a neuron $j$: \n",
    "\n",
    "$$f_{ij} = \\sum_{k}w_{ik}s_k - w_{ii}s_i - w_{ij}s_j $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The asymptotic storage capacity of Hopfield Network with Storkey learning rule is $O\\Big(\\frac{n}{\\sqrt[3]{4 \\ln n}}\\Big)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to Storkey's thesis, this rule is an approximation to a pseudoinverse rule.\n",
    "\n",
    "Using Neumann series for inverse matrix ( $(I-T)^{-1} = \\sum_i T^i$ ), one can approximate $(Z^TZ)^{-1}$ term in pseudoinverse rule (but, of course, there is more to it)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEYCAYAAABGJWFlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEKNJREFUeJzt3XuQJWV9xvHvs6xyW9gVxcLlGkWIxtL1D7NYlSipSJAQEqMiiZVENCSFiUkqhRZeykghGKMmokUilooSyRpdjQZJKVk1eLdSWmCMF7xi1qwLrOwKCCTEffNH98jZ8TczZ2bOzpwz8/1Undpz+va+3W93P/2ePtOb1hqSJE23ZrkrIEkaTwaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSmMREEmuT3LetGEtyYlLWIeHJbkmyY6+7BOWquxxNybtc2aSTyXZk2RnkrcmOWypyh9nY9I+v5TkS337/CDJ+5McvVTlj7NxaJ9pZV85bPljERCjlmTtAmbbC3wYeMaIq6NpFtg+64FLgI3Ao4CjgdeOsl7qLLB9vgKc3lrbQNdG3wDeNNKKCVhw+0zN+wvAI4aeobU2khdwM/ASuh1lN/B24KB+3IOAa4Hb+nHXAsf04y4FfgzcC9wFXA58AmjAj/ph5/TT/hpwI7AH+Azw2GnlXwj8B/A/wNp+2Av7YT8E3j1Vp1nWY21f9gmj2jbj8Fop7TOwvKcDX1ru7Wr7lOtyIPCXwFeWe7vaPvusw1rgBuCxffknzrneI96A/wkcCxwBfBq4pB/3YLor80OAw4CtwAcG5r0eOG/a8vZZAeDxwK3AZuAA4Dl9mQcOlH9jX/7BA8P+ne6K5gjgq8D5c6zHSg6IiW+fgfIuA/5xuber7bNPmcfRndz2AvcB5y73drV99inzRcAbqvJnnGfEG/D8gc+/Cnxrhmk3AbvnuQHfBLxy2jQ3AU8eKP95RZ1+Z+Dza4Ar5liPlRwQE98+/XSn0V2pnbTc29X2Ket3BN3V7inLvV1tn5+MOxb4JrC+Kn+m16jvQWwfeP9dumQjySFJ3pzku0nuoOtibUhywDyWfTxwQX8TbE+SPXQrvXGG8qfsHHh/N7BuHmWuNBPfPklOAbYAz2ytfX0e9ZsEE98+AK2124GrgH9ezPflY2iS2+cy4OLW2g/nUaeRB8SxA++PA3b07y8ATgY2t9YOB57UD0//bxti2duBS1trGwZeh7TW3jUwzTDLWc0mun2SPB64hu5K6qOLWdaYmuj2mWYt8FDg8BEuc7lNcvv8MvDa/heAU6Hy2STPnm2mUQfEHyc5JskRwMvobppA973cPcCeftwrps13C/DwOYa9BTg/yeZ0Du1/+jiynzomOYjuBhvAgf3nlWRi2yfJY+h+ZfYnrbUPjmKZY2iS2+fpSU5OsibJkcDfADf0vYmVYmLbBzgJeBzd11+b+mFnAe+fbaZRB8QW4F+BbwPfovtZInTdm4OBXcDn6A70QW8Anplkd5I39sMuAq7qu1vPaq19HvgDul8B7Kb7Pu3cEdf/HrpfFQB8rf+8kkxy+1wAHAm8Lcld/evLI1z+OJjk9jm6r9edwJfoblT/5giXPw4mtn1aa7e21nZOvfrBu1prs57j0t+wWLQkN9PdiPnISBaokbJ9xpvtM95Wa/usyD+UkyQtngEhSSqN7CsmSdLKYg9CklTaL3/Ectqasye+W7Jt79bMPdVksn3G296dj5z49llz1DdsnzE2bPvYg5AklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklcbuvwO8bseNPzXs9I2biilHM5/mx/bZ/0a9Xaptv7/L3LZ3pIubSOO8fw/bPvYgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEmlsftLaklaCYb5C/ZxZw9CklQyICRJJQNCklQau3sQC30C4jg/OXElsX2k1cMehCSpZEBIkkoGhCSpZEBIkkpLdpN6XP7LyXGpx6Qa5R//uN21ko3z/u1/OSpJWhQDQpJUMiAkSSUDQpJUWrKb1NUNm/19w9gb0otj+0irmz0ISVLJgJAklQwISVLJgJAklZb1cd/D3rge5fJVW44bxsvxwwVpqfhfjkqSViwDQpJUMiAkSSUDQpJUWjH/J7UWZ1y2+7jUQ5I9CEnSDAwISVLJgJAklQwISVIprbXlroMkaQzZg5AklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklQwISVLJgBhSkg8lec4s469I8vIhl3V9kvNGVztpvCR5aZK3LkO5LcmJS13uSrWqAyLJzUmeMsy0rbUzWmtX9fOdm+RT08af31p75QjqtCHJlUl2JrkzydeTvHhg/LwOAMNI0/X7/T1J7ur3s3ckWTfkvKcm+d5c07XWXtVaG6v9rj8W7u3Xe1eSf0rysCHmO6E/7tYODPupc8BKtKoDYky9HlgHPApYD/w68M1lrZFWorNaa+uATcDjgZeMasGDJ9Ix9IJ+vU8CNtAdb0tuzLfRTxgQvakrgiSvS7I7yXeSnDEw/vok5yV5FHAF8MT+SmRPP/4dSS7p3z8oybVJbuuXdW2SY4asyhOALa213a21va21r7XW3tsv9xP9NF/syz5ntrKSXAr8InB5P/3l/fCfTbItye1JbkryrBFsQk2g1tpO4Dq6oAAgyYH9cfBfSW7pvz49OMmhwIeAjf3+dFeSjUkuSvLeJFcnuQM4tx929cAyT0nymSR7knwxyan98HOSfH6wTkn+PMk1s9VlYNoXJfl+kh1JnjeP9b4deB/wmH45Zya5IckdSbYnuWhg8qnjbk+/zk+kPgfMWNepnleSC5PsBN4+MOyCJLf26/HcYddhKRgQ+9oM3AQ8BHgN8LYkGZygtfZV4Hzgs621da21DcVy1gBvB44HjgPuAS4fsg6fAy5N8twkj5xW9pP6t4/ry373bGW11l4GfJL+qqm19oL+IN8GbAEeCvwW8HdJHj1k/bSC9BcTZ7BvL/XVdFfYm4ATgaOBv2it/aifdke/P61rre3o5/kN4L10V+X/MK2Mo4F/AS4BjgBeCLwvyZHAB4GTp+3rz6bbP2esS7/cp/bLOg14JDDU18X9vA8BngHc0A/6EfB7ff3PBJ6f5Gn9uKnjbkO/zp+lPgfMWNfeUf36Hw/84cCw9f20vw/8bZIHDbse+11rbdW+gJuBp/TvzwW+OTDuEKABR/WfrwfOG5j2U9OW9Q7gkhnK2QTsHvj8k2UV0x4MvBT4AnAf3YF7xsD4Bpw4yzrNWhZwDvDJafO8GXjFcreHr6V59fv9XcCd/f70UbqTH0DoTpaPGJj+icB3+venAt+btryLgE8Uw67u318IvHPa+OuA5/Tvr6YLIOhO9Hf2x99cdbkSePXAuJNmOz76Y+FuYA/w33RBduQM014GvL5/f0K/3LUD4/c5Bwy53f4XOGhg/Kl0F3SDy70VOGW595Gp10R8D7aEdk69aa3d3Xcehrp5NyjJIXTfbT4VmLoaOCzJAa21H882b2vtHuBVwKuSHA68GNia5LjWdYsXW9bxwOapbnFvLfDO4ddQK8DTWmsfSfJkuqv1h9CdOI+kOzl/YaDzHOCAOZa3fZZxxwNnJzlrYNgDgH/r328B/hq4mK738IH++HvoHHXZSHchNeW7c9QR4E9baz/166okm+l6AI8BHggcCGwdYnlThtlut7XW7p023w9aa/838PluFnDO2V/8imlh2hzjLwBOBja31g7n/i5qZp6lKKS1O+jC4lDgZxZY1vS6bgc+3lrbMPBa11p7/nzqppWhtfZxut7v6/pBu+iuan9uYP9Y37obuzDzvj/bMbGdrgcxuM8d2lp7dT9+G3Bkkk3Ab3P/10tz1eX7wLED5Rw33FqXtgDXAMe21tbT3WOY6Riqhs1V15mWM9YMiIW5BTgmyQNnGH8Y3c6yJ8kRwCuGXXCSlyd5QpIHJjkI+DO6K7ubBsp++DzKmj79tcBJSX43yQP61xPS3XzX6nQZcFqSx7XW9gJvAV7fX8GT5Ogkp/fT3gI8OMn6eSz/auCsJKcnOSDJQf0N2mMAWmv30V2tv5buO/pt/fC56vIeuhvij+570kMfZ4XDgNtba/cm+Xm6nsyU24C97Hsc7XMOGKKuE8mAWJiPAV8GdibZVYy/jO5ewi66m84fnseyG91N513ADrobcGe21u7qx18EXNX/GuRZQ5T1BuCZ6X7h9MbW2p3Ar9DdnN5B97XaX9F1qbUKtdZuA/6e+2+oXkh37+tz/a+SPkLXS6W19jXgXcC3+31w4xDL3053E/uldCfb7cCL2Pf8s4XuJvPWaV+5zFaXD9Ht/x/rp/nYvFf+fn8EXJzkTrrt8J6B+t8NXAp8ul/nU6jPATPWdVKlvzEiSdI+7EFIkkoGhCSpZEBIkkoGhCSptGR/KHfamrO9G65527Z367z+dmQl8xjSQizmGLIHIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpNKSPazvuh03Lmi+0zduGnFNJEnDsAchSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSot2dNcJ8mkPHl2Uuqp1WdS9s1JqedysQchSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKk0pI97nu1PB53IRb6yGFJ2p/sQUiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSkv2NFfNbKFPul2Op8AutEyf5itNHnsQkqSSASFJKhkQkqSSASFJKhkQkqSSASFJKhkQkqSSASFJKhkQkqSSASFJKhkQkqSSASFJKhkQkqRSWmtLUtBpa85eUEE+PXT0FvMU2KV+8uyao76RBc24Ai30GNLqtm3v1gUfQ/YgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEmltctdAS29xTzpdqmfrrtt74Jm0wCfiDxeJqk97EFIkkoGhCSpZEBIkkoGhCSpZEBIkkoGhCSpZEBIkkoGhCSpZEBIkkoGhCSpZEBIkkoGhCSpZEBIkkpj/zTXhT7BcKFPTFxMmUttMeu4UJOybXQ/j6HRWy3bxh6EJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSmP/uO+FPlZ3MY/UXY7HaC+lSXrcsJaPx9DMVssxZA9CklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQa+6e5LsdTExda5nI8eVYaR+7TK4M9CElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSKa215a6DJGkM2YOQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJX+H5hB4KrItk4DAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from visualisation import run_visualisation\n",
    "learning_options = {'sc' : True, 'incremental' : True}\n",
    "retrieval_options = {'time_of_retrieval' : 100, 'sync' : False}\n",
    "options = {'rule' : 'Storkey', 'learning_options' : learning_options, 'retrieval_options' : retrieval_options}\n",
    "run_visualisation(options=options, flips = 10, pattern_num = 1, random_patterns = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradient descent and updates of weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can formulate the task of memorising a pattern as an optimisation task. Suppose we want to memorise a pattern $\\boldsymbol{\\sigma}$. Then, the optimisation task would be:\n",
    "\n",
    "$$\n",
    "\\min_{W} d \\big(\\text{sign}(W\\sigma + b), \\sigma\\big)\n",
    "$$\n",
    "with respect to similarity measure $d(x, y)$. For the sake of example, let's define it as a simple square norm: $d(x,y) = (x-y)^2$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The problem with this function is that the derivative of sign function is zero everywere apart from the origin. Which isn't great. But what if we substitute $sign(\\cdot)$ by the some differentiable monotonically increasing function $f(\\lambda\\cdot)$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the function to minimise would be: \n",
    "$$E = \\Big(f\\big(\\lambda (W\\sigma + b)\\big) - \\sigma\\Big)^2$$\n",
    "Taking a gradient of this function elementwise:\n",
    "\n",
    "$$E = \\sum_{k}z_k^2$$\n",
    "\n",
    "where $z_k = \\big(f(\\lambda h_k) - \\sigma_k\\big)$, $h_k = \\sum_{j} w_{kj}\\sigma_j + b_i$\n",
    "\n",
    "$$\n",
    "\\frac{d E}{d w_{ij}} = 2z_i\\frac{d z_i}{d x}\\Big|_{x = \\lambda h_i} = 2\\big(f(\\lambda h_i) - \\sigma_i\\big)\\frac{d f(x)}{d x}\\Big|_{x = \\lambda h_i}\\lambda\\sigma_j\n",
    "$$\n",
    "\n",
    "And thus, as the update of the weights and biases should be proportional to antigradient, to minimise the function:\n",
    "\n",
    "$$\n",
    "\\Delta w_{ij} = - 2\\lambda \\big(f(\\lambda h_i) - \\sigma_i\\big)\\frac{d f(x)}{d x}\\Big|_{x = \\lambda h_i}\\sigma_j\n",
    "$$\n",
    "$$\n",
    "\\Delta b_{i} = - 2\\lambda \\big(f(\\lambda h_i) - \\sigma_i\\big)\\frac{d f(x)}{d x}\\Big|_{x = \\lambda h_i}\n",
    "$$\n",
    "\n",
    "or in vector form:\n",
    "\n",
    "$$\n",
    "\\Delta W = - 2\\lambda \\Big(\\big(f(\\lambda h) - \\sigma\\big) \\odot \\frac{d f(x)}{d x}\\Big|_{x = \\lambda h}\\Big)\\sigma^T\n",
    "$$\n",
    "$$\n",
    "\\Delta b = - 2\\lambda \\Big(\\big(f(\\lambda h) - \\sigma\\big) \\odot \\frac{d f(x)}{d x}\\Big|_{x = \\lambda h}\\Big)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DescentL2 rule\n",
    "\n",
    "Taking the function $f(x) = x$, we could write our update rule as follows:\n",
    "\n",
    "$$\n",
    "\\Delta w_{ij} = 2\\lambda \\big(\\sigma_i - \\lambda h_i\\big)\\sigma_j\n",
    "$$\n",
    "$$\n",
    "\\Delta b_{i} = 2\\lambda \\big(\\sigma_i - \\lambda h_i\\big)\n",
    "$$\n",
    "\n",
    "Vectorised form:\n",
    "$$\n",
    "\\Delta W = 2\\lambda \\big(\\sigma - \\lambda h\\big)\\sigma^T\n",
    "$$\n",
    "$$\n",
    "\\Delta b = 2\\lambda \\big(\\sigma - \\lambda h\\big)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These two rules are somehow similar to the Storkey rule, but they are non symmetric and are more biologically plausible (as the receiving - postsinaptic - neuron requires only information available at this neuron, and the neuron doesn't have to subtract the contribution of presynaptic neuron from it's local field, as it is the case in Storkey rule). Moreover, now we know how to update biases! The intuition is simple: we have to update our biases as if they are the weighs of connections from a neuron from outside the network, which is always in +1 state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEYCAYAAABGJWFlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEJlJREFUeJzt3XuQZGV9xvHvA6vcFnZFsXS5RhGisXT9wyxWJUoqEiSExHgjsZKIhqQwMUml0MJLGSkFY9REtEjU8kokaxSNBkkpWTWoKFZKC4zxglfMmnWBlV0BgcS4b/44Z6R3/M1Mz0zvTHfP91PVtd3n9r7nvOec57x9es6mtYYkSbMdsNoVkCSNJwNCklQyICRJJQNCklQyICRJJQNCklQyICRJpbEIiCTXJDl31rCW5MQVrMODk1yZZEdf9gkrVfa4G5P2OTPJtUn2JNmZ5K1JDl+p8sfZmLTPLyX5Yt8+30/ygSRHr1T542wc2mdW2W8ftvyxCIhRS7JuCbPtBT4CPHXE1dEsS2yfDcBFwCbg4cDRwGtGWS91ltg+XwZOb61tpGujrwNvHGnFBCy5fWbm/QXgoUPP0FobyQu4CXgR3Y6yG3gHcHA/7n7AVcCt/birgGP6cRcDPwbuAe4ELgU+CTTgh/2ws/tpfw24AdgDfAZ41KzyLwD+A/gfYF0/7Pn9sB8A75mp0zzrsa4v+4RRbZtxeE1L+wws7ynAF1d7u9o+5bocBPwl8OXV3q62zz7rsA64HnhUX/6JC673iDfgfwLHAkcCnwYu6sfdn+7K/FDgcOAK4IMD814DnDtrefusAPAY4BZgC3Ag8Ky+zIMGyr+hL/+QgWH/TndFcyTwFeC8BdZjmgNi4ttnoLxLgH9c7e1q++xT5nF0J7e9wI+Ac1Z7u9o++5T5AuD1VflzzjPiDXjewOdfBb45x7Sbgd2L3IBvBF4xa5obgScMlP+cok6/M/D51cCbFliPaQ6IiW+ffrrT6K7UTlrt7Wr7lPU7ku5q95TV3q62z0/GHQt8A9hQlT/Xa9T3ILYPvP8OXbKR5NAkb07ynSS303WxNiY5cBHLPh44v78JtifJHrqV3jRH+TN2Dry/C1i/iDKnzcS3T5JTgK3A01prX1tE/SbBxLcPQGvtNuAy4J+X8335GJrk9rkEeHlr7QeLqNPIA+LYgffHATv69+cDJwNbWmtHAI/vh6f/tw2x7O3Axa21jQOvQ1tr7x6YZpjlrGUT3T5JHgNcSXcl9bHlLGtMTXT7zLIOeCBwxAiXudomuX1+GXhN/wvAmVC5Lskz55tp1AHxx0mOSXIk8BK6mybQfS93N7CnH/eyWfPdDDxkgWFvAc5LsiWdw/qfPo7sp45JDqa7wQZwUP95mkxs+yR5JN2vzP6ktfahUSxzDE1y+zwlyclJDkhyFPA3wPV9b2JaTGz7ACcBj6b7+mtzP+ws4APzzTTqgNgK/CvwLeCbdD9LhK57cwiwC/gs3YE+6PXA05LsTvKGftiFwGV9d+sZrbXPAX9A9yuA3XTfp50z4vrfTferAoCv9p+nySS3z/nAUcDbktzZv740wuWPg0lun6P7et0BfJHuRvVvjnD542Bi26e1dktrbefMqx+8q7U27zku/Q2LZUtyE92NmI+OZIEaKdtnvNk+422tts9U/qGcJGn5DAhJUmlkXzFJkqaLPQhJUmm//BHLaQc8feK7Jdv2XpGFp5pMts9427vzYRPfPgc86Ou2zxgbtn3sQUiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSmP33wFeveOGnxp2+qbNxZSjmU+LY/vsf6PeLtW2399lbts70sVNpHHev4dtH3sQkqSSASFJKhkQkqSSASFJKhkQkqSSASFJKhkQkqSSASFJKhkQkqTS2P0ltSRNg2H+gn3c2YOQJJUMCElSyYCQJJXG7h7EUp+AOM5PTpwmto+0dtiDkCSVDAhJUsmAkCSVDAhJUmnFblKPy385OS71mFSj/OMft7um2Tjv3/6Xo5KkZTEgJEklA0KSVDIgJEmlFbtJXd2w2d83jL0hvTy2j7S22YOQJJUMCElSyYCQJJUMCElSaVUf9z3sjetRLl+11bhhvBo/XJBWiv/lqCRpahkQkqSSASFJKhkQkqTS1Pyf1Fqecdnu41IPSfYgJElzMCAkSSUDQpJUMiAkSaW01la7DpKkMWQPQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDQpJUMiAkSSUDYkhJPpzkWfOMf1OSlw65rGuSnDu62knjJcmLk7x1FcptSU5c6XKn1ZoOiCQ3JXniMNO21s5orV3Wz3dOkmtnjT+vtfaKEdRpY5K3J9mZ5I4kX0vywoHxizoADCPN1u/3dye5s9/P3plk/ZDznprkuwtN11p7ZWttrPa7/li4p1/vXUn+KcmDh5jvhP64Wzcw7KfOAdNoTQfEmHodsB54OLAB+HXgG6taI02js1pr64HNwGOAF41qwYMn0jH0vH69TwI20h1vK27Mt9FPGBC9mSuCJK9NsjvJt5OcMTD+miTnJnk48Cbgcf2VyJ5+/DuTXNS/v1+Sq5Lc2i/rqiTHDFmVxwJbW2u7W2t7W2tfba29r1/uJ/tpvtCXffZ8ZSW5GPhF4NJ++kv74T+bZFuS25LcmOQZI9iEmkCttZ3A1XRBAUCSg/rj4L+S3Nx/fXpIksOADwOb+v3pziSbklyY5H1JLk9yO3BOP+zygWWekuQzSfYk+UKSU/vhZyf53GCdkvx5kivnq8vAtC9I8r0kO5I8ZxHrfRvwfuCR/XLOTHJ9ktuTbE9y4cDkM8fdnn6dH0d9DpizrjM9ryQXJNkJvGNg2PlJbunX49nDrsNKMCD2tQW4EXgA8GrgbUkyOEFr7SvAecB1rbX1rbWNxXIOAN4BHA8cB9wNXDpkHT4LXJzk2UkeNqvsx/dvH92X/Z75ymqtvQT4FP1VU2vtef1Bvg3YCjwQ+C3g75I8Ysj6aYr0FxNnsG8v9VV0V9ibgROBo4G/aK39sJ92R78/rW+t7ejn+Q3gfXRX5f8wq4yjgX8BLgKOBJ4PvD/JUcCHgJNn7evPpNs/56xLv9wn9cs6DXgYMNTXxf28DwCeClzfD/oh8Ht9/c8Enpvkyf24meNuY7/O11GfA+asa+9B/fofD/zhwLAN/bS/D/xtkvsNux77XWttzb6Am4An9u/PAb4xMO5QoAEP6j9fA5w7MO21s5b1TuCiOcrZDOwe+PyTZRXTHgK8GPg88CO6A/eMgfENOHGedZq3LOBs4FOz5nkz8LLVbg9fK/Pq9/s7gTv6/eljdCc/gNCdLB86MP3jgG/3708FvjtreRcCnyyGXd6/vwB416zxVwPP6t9fThdA0J3o7+iPv4Xq8nbgVQPjTprv+OiPhbuAPcB/0wXZUXNMewnwuv79Cf1y1w2M3+ccMOR2+1/g4IHxp9Jd0A0u9xbglNXeR2ZeE/E92AraOfOmtXZX33kY6ubdoCSH0n23+SRg5mrg8CQHttZ+PN+8rbW7gVcCr0xyBPBC4Iokx7WuW7zcso4Htsx0i3vrgHcNv4aaAk9urX00yRPortYfQHfiPIru5Pz5gc5zgAMXWN72ecYdDzw9yVkDw+4D/Fv/fivw18DL6XoPH+yPvwcuUJdNdBdSM76zQB0B/rS19lO/rkqyha4H8EjgvsBBwBVDLG/GMNvt1tbaPbPm+35r7f8GPt/FEs45+4tfMS1NW2D8+cDJwJbW2hHc20XN3LMUhbR2O11YHAb8zBLLml3X7cAnWmsbB17rW2vPXUzdNB1aa5+g6/2+th+0i+6q9ucG9o8NrbuxC3Pv+/MdE9vpehCD+9xhrbVX9eO3AUcl2Qz8Nvd+vbRQXb4HHDtQznHDrXVpK3AlcGxrbQPdPYa5jqFq2EJ1nWs5Y82AWJqbgWOS3HeO8YfT7Sx7khwJvGzYBSd5aZLHJrlvkoOBP6O7srtxoOyHLKKs2dNfBZyU5HeT3Kd/PTbdzXetTZcApyV5dGttL/AW4HX9FTxJjk5yej/tzcD9k2xYxPIvB85KcnqSA5Mc3N+gPQagtfYjuqv119B9R7+tH75QXd5Ld0P8EX1PeujjrHA4cFtr7Z4kP0/Xk5lxK7CXfY+jfc4BQ9R1IhkQS/Nx4EvAziS7ivGX0N1L2EV30/kji1h2o7vpvAvYQXcD7szW2p39+AuBy/pfgzxjiLJeDzwt3S+c3tBauwP4Fbqb0zvovlb7K7outdag1tqtwN9z7w3VC+jufX22/1XSR+l6qbTWvgq8G/hWvw9uGmL52+luYr+Y7mS7HXgB+55/ttLdZL5i1lcu89Xlw3T7/8f7aT6+6JW/1x8BL09yB912eO9A/e8CLgY+3a/zKdTngDnrOqnS3xiRJGkf9iAkSSUDQpJUMiAkSSUDQpJUWrE/lDvtgKdP/d3wq3fcsKT5Tt+0eeGJ1qhte69Y1N+OTLO1cAxp9JZzDNmDkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUmnFHta30pb64DxY+sPz1sJD93wgobR22IOQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJWm9mmuy3l6qE8sndtaWEct36QcQ5NSz9ViD0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEmlqX3c91rgo4ol7U/2ICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJpRV7mutaePLoUtdRksaRPQhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUmnFnuY6SU9lnaS6SuNoUo6hSannarEHIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqrdjTXJfq6h03LGm+tfCUxqVum9WwFtpD95qkfXMp1sr+bA9CklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQa+6e5LvWpict5muSkPKnRbaP9yf1kbmtl29iDkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUmnsH/e9VMt5pO5yHuU7CSbpccNavqXuz+4nc1uN88tqtIc9CElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSaWqf5rocPsVybpP0JEp13PbjZZLawx6EJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKmU1tpq10GSNIbsQUiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSv8PS8DTXb7O1VcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from visualisation import run_visualisation\n",
    "learning_options = {'sc' : False, 'incremental' : True, 'tol' : 1e-2, 'lmbd' : 0.5, 'alpha' : 0.01}\n",
    "retrieval_options = {'time_of_retrieval' : 100, 'sync' : True}\n",
    "options = {'rule' : 'DescentL2', 'learning_options' : learning_options, 'retrieval_options' : retrieval_options}\n",
    "run_visualisation(options=options, flips = 10, pattern_num = 1, random_patterns = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hebbian learning rule revisited\n",
    "Let's now change the similarity measure between two vectors:\n",
    "$$ d(x,y) = -(x,y)$$\n",
    "so that if two vectors are aligned parallely they achieve minimum\n",
    "\n",
    "We need to minimise function $E = -\\big(f(\\lambda W \\sigma), \\sigma\\big)$\n",
    "\n",
    "$E = -\\sum_k z_k$, where $z_k = f(\\lambda h_k) \\sigma_k$, and, thus\n",
    "$$\n",
    "\\frac{d E}{d w_{ij}} = -\\frac{d z_i}{d x}\\Big|_{x = \\lambda h_i} = -\\frac{d f(x)}{d x}\\Big|_{x = \\lambda h_i} \\lambda \\sigma_j \\sigma_i\n",
    "$$\n",
    "\n",
    "And the update of weights to minimise the function would be:\n",
    "\n",
    "$$\n",
    "\\Delta w_{ij} = \\lambda \\frac{d f(x)}{d x}\\Big|_{x = \\lambda h_i} \\sigma_j \\sigma_i\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\Delta W = \\lambda \\big(\\frac{d f(x)}{d x}\\Big|_{x = \\lambda h} \\odot \\sigma\\big) \\sigma^T\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if $f(x) = x$, then\n",
    "$$\n",
    "\\Delta w_{ij} = \\lambda \\sigma_i \\sigma_j\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\Delta W = \\lambda \\sigma \\sigma^T\n",
    "$$ \n",
    "Which is exactly what Hebbian learning does!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparison of the performance of various rules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='../imgs_for_jupyter/flips_and_patterns_Hebb_sc_100.png' width=\"1000\" height=\"1000\" align=\"center\"/>\n",
    "<img src='../imgs_for_jupyter/flips_and_patterns_pseudoinverse_sc_100.png' width=\"1000\" height=\"1000\" align=\"center\"/>\n",
    "<img src='../imgs_for_jupyter/flips_and_patterns_Storkey_sc_100.png' width=\"1000\" height=\"1000\" align=\"center\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEYCAYAAABGJWFlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEHlJREFUeJzt3XuQJWV9xvHvA6vcFlhRLF2uUYRoLF3/MItViZKKBAkhMd5IrCSiISlMTFIptPBSRkrBGDURLRK1vBIJRtFokJSSVYN3K6UFxnjBK2bNusDKroBAYtw3f/Q7cnb8zeyZmcPMmdnvp+rUntO39+1+u/vp9/SZ3rTWkCRptv1WugKSpOlkQEiSSgaEJKlkQEiSSgaEJKlkQEiSSgaEJKk0FQGR5Jok58wa1pKcsIx1eGCSK5Ns62Ufv1xlT7spaZ8zknwyya4k25O8Ocmhy1X+NJuS9vmlJF/s7fP9JO9LctRylT/NpqF9ZpX91nHLn4qAmLQk6xYx227gQ8CTJ1wdzbLI9jkcuBDYCDwUOAp41STrpcEi2+fLwGmttQ0MbfR14PUTrZiARbfPzLy/ADx47BlaaxN5ATcAL2DYUXYCbwMO7OPuA1wF3NzHXQUc3cddBPwYuAu4HbgE+DjQgB/2YWf1aX8NuA7YBXwaeMSs8s8H/gP4H2BdH/bcPuwHwLtm6jTPeqzrZR8/qW0zDa+10j4jy3sS8MWV3q62T7kuBwB/CXx5pber7bPHOqwDrgUe0cs/Ya/rPeEN+J/AMcARwKeAC/u4+zJcmR8MHApcAbx/ZN5rgHNmLW+PFQAeBdwEbAb2B57RyzxgpPzrevkHjQz7d4YrmiOArwDn7mU91nJArPr2GSnvYuAfV3q72j57lHksw8ltN/Aj4OyV3q62zx5lPg94bVX+nPNMeAOeO/L5V4FvzjHtJmDnAjfg64GXzZrmeuBxI+U/q6jT74x8fiXwhr2sx1oOiFXfPn26Uxmu1E5c6e1q+5T1O4Lhavfkld6uts9Pxh0DfAM4vCp/rtek70FsHXn/HYZkI8nBSd6Y5DtJbmXoYm1Isv8Cln0ccF6/CbYryS6Gld44R/kzto+8vwNYv4Ay15pV3z5JTgYuB57SWvvaAuq3Gqz69gFord0CXAr881K+L59Cq7l9LgZe2lr7wQLqNPGAOGbk/bHAtv7+POAkYHNr7TDgsX14+r9tjGVvBS5qrW0YeR3cWnvnyDTjLGdftqrbJ8mjgCsZrqQ+spRlTalV3T6zrAPuDxw2wWWutNXcPr8MvKr/AnAmVD6T5OnzzTTpgPjjJEcnOQJ4EcNNExi+l7sT2NXHvWTWfDcCD9rLsDcB5ybZnMEh/aePE/upY5IDGW6wARzQP68lq7Z9kjyc4Vdmf9Ja+8AkljmFVnP7PCnJSUn2S3Ik8DfAtb03sVas2vYBTgQeyfD116Y+7EzgffPNNOmAuBz4V+BbwDcZfpYIQ/fmIGAH8FmGA33Ua4GnJNmZ5HV92AXApb279bTW2ueAP2D4FcBOhu/Tzp5w/e9k+FUBwFf757VkNbfPecCRwFuS3N5fX5rg8qfBam6fo3q9bgO+yHCj+jcnuPxpsGrbp7V2U2tt+8yrD97RWpv3HJd+w2LJktzAcCPmwxNZoCbK9pluts9021fbZ03+oZwkaekMCElSaWJfMUmS1hZ7EJKk0j3yRyyn7vfUVd8t2bL7iux9qtXJ9pluu7c/ZNW3z34P+LrtM8XGbR97EJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKkkgEhSSoZEJKk0tT9d4BXb7vup4adtnFTMeVk5tPC2D73vElvl2rb39Nlbtk90cWtStO8f4/bPvYgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEklA0KSVDIgJEmlqftLaklaC8b5C/ZpZw9CklQyICRJJQNCklSaunsQi30C4jQ/OXEtsX2kfYc9CElSyYCQJJUMCElSyYCQJJWW7Sb1tPyXk9NSj9Vqkn/843bXWjbN+7f/5agkaUkMCElSyYCQJJUMCElSadluUlc3bO7pG8bekF4a20fat9mDkCSVDAhJUsmAkCSVDAhJUmlFH/c97o3rSS5ftZW4YbwSP1yQlov/5agkac0yICRJJQNCklQyICRJpTXzf1JraaZlu09LPSTZg5AkzcGAkCSVDAhJUsmAkCSV0lpb6TpIkqaQPQhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDIgxJflgkmfMM/4NSV485rKuSXLO5GonTZckL0zy5hUotyU5YbnLXav26YBIckOSx48zbWvt9NbapX2+s5N8ctb4c1trL5tAnTYkeWuS7UluS/K1JM8fGb+gA8Aw0mx9v78zye19P3t7kvVjzntKku/ubbrW2stba1O13/Vj4a6+3juS/FOSB44x3/H9uFs3MuynzgFr0T4dEFPqNcB64KHA4cCvA99Y0RppLTqztbYe2AQ8CnjBpBY8eiKdQs/p630isIHheFt2U76NfsKA6GauCJK8OsnOJN9OcvrI+GuSnJPkocAbgMf0K5Fdffzbk1zY398nyVVJbu7LuirJ0WNW5dHA5a21na213a21r7bW3tOX+/E+zRd62WfNV1aSi4BfBC7p01/Sh/9ski1JbklyfZKnTWATahVqrW0HrmYICgCSHNCPg/9KcmP/+vSgJIcAHwQ29v3p9iQbk1yQ5D1JLktyK3B2H3bZyDJPTvLpJLuSfCHJKX34WUk+N1qnJH+e5Mr56jIy7fOSfC/JtiTPWsB63wK8F3h4X84ZSa5NcmuSrUkuGJl85rjb1df5MdTngDnrOtPzSnJ+ku3A20aGnZfkpr4ezxx3HZaDAbGnzcD1wP2AVwJvSZLRCVprXwHOBT7TWlvfWttQLGc/4G3AccCxwJ3AJWPW4bPARUmemeQhs8p+bH/7yF72u+Yrq7X2IuAT9Kum1tpz+kG+BbgcuD/wW8DfJXnYmPXTGtIvJk5nz17qKxiusDcBJwBHAX/RWvthn3Zb35/Wt9a29Xl+A3gPw1X5P8wq4yjgX4ALgSOA5wLvTXIk8AHgpFn7+tMZ9s8569KX+4S+rFOBhwBjfV3c570f8GTg2j7oh8Dv9fqfATw7yRP7uJnjbkNf589QnwPmrGv3gL7+xwF/ODLs8D7t7wN/m+Q+467HPa61ts++gBuAx/f3ZwPfGBl3MNCAB/TP1wDnjEz7yVnLejtw4RzlbAJ2jnz+ybKKaQ8CXgh8HvgRw4F7+sj4BpwwzzrNWxZwFvCJWfO8EXjJSreHr+V59f3+duC2vj99hOHkBxCGk+WDR6Z/DPDt/v4U4LuzlncB8PFi2GX9/fnAO2aNvxp4Rn9/GUMAwXCiv60ff3ury1uBV4yMO3G+46MfC3cAu4D/ZgiyI+eY9mLgNf398X2560bG73EOGHO7/S9w4Mj4Uxgu6EaXexNw8krvIzOvVfE92DLaPvOmtXZH7zyMdfNuVJKDGb7bfAIwczVwaJL9W2s/nm/e1tqdwMuBlyc5DHg+cEWSY9vQLV5qWccBm2e6xd064B3jr6HWgCe21j6c5HEMV+v3YzhxHslwcv78SOc5wP57Wd7WecYdBzw1yZkjw+4F/Ft/fznw18BLGXoP7+/H3/33UpeNDBdSM76zlzoC/Glr7ad+XZVkM0MP4OHAvYEDgCvGWN6Mcbbbza21u2bN9/3W2v+NfL6DRZxz7il+xbQ4bS/jzwNOAja31g7j7i5q5p6lKKS1WxnC4hDgZxZZ1uy6bgU+1lrbMPJa31p79kLqprWhtfYxht7vq/ugHQxXtT83sn8c3oYbuzD3vj/fMbGVoQcxus8d0lp7RR+/BTgyySbgt7n766W91eV7wDEj5Rw73lqXLgeuBI5prR3OcI9hrmOoGra3us61nKlmQCzOjcDRSe49x/hDGXaWXUmOAF4y7oKTvDjJo5PcO8mBwJ8xXNldP1L2gxZQ1uzprwJOTPK7Se7VX4/OcPNd+6aLgVOTPLK1tht4E/CafgVPkqOSnNanvRG4b5LDF7D8y4Azk5yWZP8kB/YbtEcDtNZ+xHC1/iqG7+i39OF7q8u7GW6IP6z3pMc+zgqHAre01u5K8vMMPZkZNwO72fM42uMcMEZdVyUDYnE+CnwJ2J5kRzH+YoZ7CTsYbjp/aAHLbgw3nXcA2xhuwJ3RWru9j78AuLT/GuRpY5T1WuApGX7h9LrW2m3ArzDcnN7G8LXaXzF0qbUPaq3dDPw9d99QPZ/h3tdn+6+SPszQS6W19lXgncC3+j64cYzlb2W4if1ChpPtVuB57Hn+uZzhJvMVs75yma8uH2TY/z/ap/noglf+bn8EvDTJbQzb4d0j9b8DuAj4VF/nk6nPAXPWdbVKvzEiSdIe7EFIkkoGhCSpZEBIkkoGhCSptGx/KHfqfk/1brgWbMvuKxb0tyNrmceQFmMpx5A9CElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJUMCElSyYCQJJWW7WF9mh5Xb7tu0fOetnHTBGsiaZrZg5AklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAklQwISVLJgJAkldbs01yX8sTSxVruJ50udh19IqvGsRLHkKaLPQhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSVDAhJUsmAkCSV1uzjvpfySGsfcyxJ9iAkSXMwICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklRas09z9Yms0tIs5YnImh5bdi9+XnsQkqSSASFJKhkQkqSSASFJKhkQkqSSASFJKhkQkqSSASFJKhkQkqSSASFJKhkQkqSSASFJKhkQkqTSmn2a677wJMrFruNSnnS7EmVKWhn2ICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJpal/mutinwK6LzzNdbGWsm2Wuz227F7UbJoAn8ArexCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpNLUP83Vp7JOnk/plDQOexCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqTf3jvlfCYh+HvdhHky93eUt5hLqPCt93+Kj9tWHL7sXPaw9CklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklQyICRJJQNCklTyaa6F5X6K5Wp6auZqqqukpbEHIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqpbW20nWQJE0hexCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpJIBIUkqGRCSpNL/A83mxG9mYkQWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from visualisation import run_visualisation\n",
    "learning_options = {'sc' : False, 'incremental' : True, 'tol' : 1e-2, 'lmbd' : 0.5}\n",
    "retrieval_options = {'time_of_retrieval' : 100, 'sync' : False}\n",
    "options = {'rule' : 'DescentExpBarrierSI', 'learning_options' : learning_options, 'retrieval_options' : retrieval_options}\n",
    "run_visualisation(options=options, flips = 10, pattern_num = 1, random_patterns = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEYCAYAAABGJWFlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEa9JREFUeJzt3X2wXVV5x/HvE6K8BRIiYTThrYpQLCPhDxqYaZVOTZFSWqu8tE5b0NIOtradDjIojoVBoFSsgEMrjspLpbEarBbpIA1SRBSmowOWqmBRobExQCSRhJeWmtU/9rpwcnnuvfuec3Je7v1+Zs7k3L332Wvtvfbev7POOmcnSilIkjTZgmFXQJI0mgwISVLKgJAkpQwISVLKgJAkpQwISVLKgJAkpUYiICLijog4c9K0EhGHDLAOr4iImyJiQy374EGVPepGpH1OjIi7ImJLRGyMiI9HxF6DKn+UjUj7/FJE3F/b58cR8bmIWDGo8kfZKLTPpLKvaVv+SAREv0XEwi5eth34IvCWPldHk3TZPouBi4DlwOHACuCyftZLjS7b59vA8aWUJTRt9J/AR/paMQFdt8/Ea38BeFXrF5RS+vIAHgbeQ3OgbAauBXar8/YBbgYer/NuBvav8y4Gfgo8C2wDrgLuBArwVJ12Wl3214D7gC3A14DXTir/XODfgf8BFtZp76rTfgJ8eqJO02zHwlr2wf3aN6PwmCvt07G+NwP3D3u/2j7ptuwK/CXw7WHvV9tnh21YCNwLvLaWf8iM293nHfgfwAHAUuCrwEV13sto3pnvAewFrAU+3/HaO4AzJ61vhw0AjgIeA1YBuwCn1zJ37Sj/vlr+7h3T/o3mHc1S4DvAWTNsx1wOiLFvn47yrgD+Ydj71fbZocwDaS5u24HngDOGvV9tnx3KPAe4Mit/ytf0eQee1fH3rwLfm2LZlcDmWe7AjwDvn7TMg8DrO8p/e1Kn3+n4+wPA1TNsx1wOiLFvn7rcapp3aocOe7/aPmn9ltK82z1m2PvV9nl+3gHAQ8DirPypHv0eg1jf8fwRmmQjIvaIiI9GxCMR8SRNF2tJROwyi3UfBJxdB8G2RMQWmo1ePkX5EzZ2PH8aWDSLMueasW+fiDgGWAOcXEr57izqNw7Gvn0ASilPANcD/9TL5+UjaJzb5wrgwlLKT2ZRp74HxAEdzw8ENtTnZwOHAatKKXsDr6vTo/5bWqx7PXBxKWVJx2OPUsqnOpZps575bKzbJyKOAm6ieSf1pV7WNaLGun0mWQjsB+zdx3UO2zi3zy8Dl9VvAE6Eyt0R8dbpXtTvgPjjiNg/IpYC76UZNIHmc7lngC113vmTXvco8MoZpn0MOCsiVkVjz/rVx7591TEidqMZYAPYtf49l4xt+0TEETTfMvuTUsoX+rHOETTO7fPmiDgsIhZExDLgQ8C9tTcxV4xt+wCHAkfSfPy1sk47CfjcdC/qd0CsAf4F+D7wPZqvJULTvdkd2ATcQ3Oid7oSODkiNkfEh+u0C4Dra3fr1FLK14E/oPkWwGaaz9PO6HP9n6H5VgHAA/XvuWSc2+dsYBnwiYjYVh/f6uP6R8E4t8+KWq+twP00A9W/2cf1j4KxbZ9SymOllI0Tjzp5Uyll2mtc1AGLnkXEwzQDMbf1ZYXqK9tntNk+o22+ts+c/KGcJKl3BoQkKdW3j5gkSXOLPQhJUmqn/Ihl9YJTWnVLbt1wX1frP375yhdNy9aVLdfWuu1rY+alxlMv7dPtPrV92mvbPm1N3ve97Pe2bJ/utb0u9nIdbNs+9iAkSSkDQpKUMiAkSSkDQpKUGrk7LXY7gNbLgE23g+VqbxADo9Kw9PNLGG1f1+8vfmTsQUiSUgaEJCllQEiSUgaEJCk1sEHqYfwqt+36818azr5e46Lf+6/NL3UHMaA233Tbjr18KcM2m19farEHIUlKGRCSpJQBIUlKGRCSpNTABqn7ObjV718azqdBJ+j/QGM/1+dg9s4/Hj0HetPv/TeM27G3ZQ9CkpQyICRJKQNCkpQyICRJqaH+krqtYQyCzuVfUrc1jFsYj9IA3bAMYxDZgWtl7EFIklIGhCQpZUBIklJj+UO5XoxKPeaSbu/m2tZ8a7O2+6rb/dzLeJDjErlejtFu77Y7iLsm24OQJKUMCElSyoCQJKUMCElSamCD1Jlh/Dek6u2/HO12P9sW7fX7bsXd6mWw1B+a5vr5JYJBfGHAHoQkKWVASJJSBoQkKWVASJJSQ72b6zD+G1KNziCo2huVtvDX1b21xbhd8+xBSJJSBoQkKWVASJJSBoQkKTXU2313O9jTyyDRqAz2DVO/BxXb3O67l3rMt/YZhrZt4X/ZOzrH4yCuZfYgJEkpA0KSlDIgJEkpA0KSlNopg9TDGAje2f+P71zXy37ptm1ti1y/z582XyIYlYHXcTWMa57/J7UkaWgMCElSyoCQJKUMCElSKkopw66DJGkE2YOQJKUMCElSyoCQJKUMCElSyoCQJKUMCElSyoCQJKUMCElSyoCQJKUMCElSyoCQJKUMCElSyoCQJKUMCElSyoCQJKUMCElSyoCQJKUMCElSyoCQJKUMCElSyoCQJKUMCElSyoCQJKUMCElSyoCQJKUMCElSyoCQJKUMCElSyoBoKSJuiYjTp5l/dUS8r+W67oiIM/tXO2m0RMR5EfHxIZRbIuKQQZc7V83rgIiIhyPiDW2WLaWcUEq5vr7ujIi4a9L8s0op7+9DnZZExDURsTEitkbEdyPi3R3zZ3UCGEaarB73z0TEtnqcXRcRi1q+9riI+OFMy5VSLimljNRxV8+FZ+t2b4qIf4yIV7R43cH1vFvYMe1F14C5aF4HxIi6HFgEHA4sBn4deGioNdJcdFIpZRGwEjgKeE+/Vtx5IR1B76zbfSiwhOZ8G7gR30fPMyCqiXcEEfHBiNgcET+IiBM65t8REWdGxOHA1cCx9Z3Iljr/uoi4qD7fJyJujojH67pujoj9W1blaGBNKWVzKWV7KeWBUsqNdb131mW+Wcs+bbqyIuJi4BeBq+ryV9XpPxsR6yLiiYh4MCJO7cMu1BgqpWwEbqUJCgAiYtd6HvxXRDxaPz7dPSL2BG4BltfjaVtELI+ICyLixoi4ISKeBM6o027oWOcxEfG1iNgSEd+MiOPq9NMi4uuddYqIP4+Im6arS8ey50TEjyJiQ0S8fRbb/QTwWeCIup4TI+LeiHgyItZHxAUdi0+cd1vqNh9Lfg2Ysq4TPa+IODciNgLXdkw7OyIeq9vxtrbbMAgGxI5WAQ8C+wIfAD4REdG5QCnlO8BZwN2llEWllCXJehYA1wIHAQcCzwBXtazDPcDFEfG2iHj1pLJfV58eWcv+9HRllVLeC3yF+q6plPLOepKvA9YA+wG/BfxtRLymZf00h9Q3EyewYy/1Upp32CuBQ4AVwF+UUp6qy26ox9OiUsqG+prfAG6keVf+95PKWAH8M3ARsBR4F/DZiFgGfAE4bNKx/laa43PKutT1vrGuazXwaqDVx8X1tfsCbwHurZOeAn6v1v9E4B0R8aY6b+K8W1K3+W7ya8CUda1eXrf/IOAPO6Ytrsv+PvA3EbFP2+3Y6Uop8/YBPAy8oT4/A3ioY94eQAFeXv++AzizY9m7Jq3rOuCiKcpZCWzu+Pv5dSXL7g6cB3wDeI7mxD2hY34BDplmm6YtCzgN+Mqk13wUOH/Y7eFjMI963G8Dttbj6Us0Fz+AoLlYvqpj+WOBH9TnxwE/nLS+C4A7k2k31OfnAp+cNP9W4PT6/AaaAILmQr+1nn8z1eUa4NKOeYdOd37Uc+FpYAvw3zRBtmyKZa8ALq/PD67rXdgxf4drQMv99r/Abh3zj6N5Q9e53seAY4Z9jEw8xuJzsAHaOPGklPJ07Ty0GrzrFBF70Hy2+UZg4t3AXhGxSynlp9O9tpTyDHAJcElE7A28G1gbEQeWplvca1kHAasmusXVQuCT7bdQc8CbSim3RcTrad6t70tz4VxGc3H+RkfnOYBdZljf+mnmHQScEhEndUx7CfCv9fka4K+BC2l6D5+v599+M9RlOc0bqQmPzFBHgD8tpbzo21URsYqmB3AE8FJgV2Bti/VNaLPfHi+lPDvpdT8upfxfx99P08U1Z2fxI6bulBnmnw0cBqwqpezNC13UmPolSSGlPEkTFnsCP9NlWZPruh74cillScdjUSnlHbOpm+aGUsqXaXq/H6yTNtG8q/25juNjcWkGdmHqY3+6c2I9TQ+i85jbs5RyaZ2/DlgWESuB3+aFj5dmqsuPgAM6yjmw3Van1gA3AQeUUhbTjDFMdQ5l02aq61TrGWkGRHceBfaPiJdOMX8vmoNlS0QsBc5vu+KIeF9EHB0RL42I3YA/o3ln92BH2a+cRVmTl78ZODQifjciXlIfR0cz+K756QpgdUQcWUrZDnwMuLy+gyciVkTE8XXZR4GXRcTiWaz/BuCkiDg+InaJiN3qAO3+AKWU52jerV9G8xn9ujp9prp8hmZA/DW1J936PEvsBTxRSnk2In6epicz4XFgOzueRztcA1rUdSwZEN25HfgWsDEiNiXzr6AZS9hEM+j8xVmsu9AMOm8CNtAMwJ1YStlW518AXF+/DXJqi7KuBE6O5htOHy6lbAV+hWZwegPNx2p/RdOl1jxUSnkc+DteGFA9l2bs6576raTbaHqplFIeAD4FfL8eg8tbrH89zSD2eTQX2/XAOex4/VlDM8i8dtJHLtPV5Raa4//2uszts974F/wRcGFEbKXZD5/pqP/TwMXAV+s2H0N+DZiyruMq6sCIJEk7sAchSUoZEJKklAEhSUoZEJKk1MB+KLd6wSkDHQ2/dcN9gyyuJ8cvXznzQolut7Hb8oZh3fa1s/rtyFzmOTQ1z6Gp9XIO2YOQJKUMCElSyoCQJKUMCElSyoCQJKUMCElSyoCQJKUMCElSyoCQJKUMCElSyoCQJKUMCElSamA36xsn43IjrkHfoGxYZWr8eA6NVpndsgchSUoZEJKklAEhSUoZEJKklAEhSUoZEJKklAEhSUoZEJKklAEhSUoZEJKklAEhSUoZEJKklAEhSUpFKWUgBa1ecEpXBXV7B8NxuZskzI9t7Na67Wtj2HUYFZ5DU5sP29itXs4hexCSpJQBIUlKGRCSpJQBIUlKGRCSpJQBIUlKGRCSpJQBIUlKGRCSpJQBIUlKGRCSpJQBIUlKGRCSpJQBIUlKLRx2BWYyjNvxzvVbB3e7fdD9NvZSpnozLsdlLwa9jfPlHLIHIUlKGRCSpJQBIUlKGRCSpJQBIUlKGRCSpJQBIUlKGRCSpJQBIUlKGRCSpJQBIUlKGRCSpJQBIUlKDexuroO+Q+o43T10nO62Oeh2XLe9q5epg+fCaBmnc8gehCQpZUBIklIGhCQpZUBIklIGhCQpZUBIklIGhCQpZUBIklIGhCQpZUBIklIGhCQpZUBIklIGhCQpNbC7uQ76rqzz4a6Qw+B+HT+22WgZp/awByFJShkQkqSUASFJShkQkqSUASFJShkQkqSUASFJShkQkqSUASFJShkQkqSUASFJShkQkqSUASFJSg3sbq7dGvRdYIdR5qDv7jhOd5PU8AzjHBq0cTlnh8UehCQpZUBIklIGhCQpZUBIklIGhCQpZUBIklIGhCQpZUBIklIGhCQpZUBIklIGhCQpZUBIklIGhCQpNfJ3cx2nuy2Oyx0e58NdOvWCcTqHBm0+bGMv7EFIklIGhCQpZUBIklIGhCQpZUBIklIGhCQpZUBIklIGhCQpZUBIklIGhCQpZUBIklIGhCQpZUBIklIGhCQpNfK3++72drze0npqvWyft44eP/PhHBqn43Kc6moPQpKUMiAkSSkDQpKUMiAkSSkDQpKUMiAkSSkDQpKUMiAkSSkDQpKUMiAkSSkDQpKUMiAkSSkDQpKUGtjdXMfpDoaD1stdM7vRyz4dxp1B1fAc6r9hHJfjdA7Zg5AkpQwISVLKgJAkpQwISVLKgJAkpQwISVLKgJAkpQwISVLKgJAkpQwISVLKgJAkpQwISVLKgJAkpaKUMuw6SJJGkD0ISVLKgJAkpQwISVLKgJAkpQwISVLKgJAkpQwISVLKgJAkpQwISVLKgJAkpQwISVLKgJAkpQwISVLKgJAkpQwISVLKgJAkpQwISVLKgJAkpQwISVLKgJAkpQwISVLKgJAkpQwISVLq/wG7csTQqfkhWAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from visualisation import run_visualisation\n",
    "learning_options = {'sc' : True, 'tol' : 1e-1, 'lr' : 1e-3} #,\n",
    "retrieval_options = {'time_of_retrieval' : 100, 'sync' : True}\n",
    "options = {'rule' : 'DiederichOpperII', 'learning_options' : learning_options, 'retrieval_options' : retrieval_options}\n",
    "run_visualisation(options=options, flips = 10, pattern_num = 1, random_patterns = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}