boltzmann-machine/Hopfield_network.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Hopfield_network.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyP/QXbTfNfl7Jr50/avFqSq",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/wenjunsun/personal-machine-learning-projects/blob/master/boltzmann-machine/Hopfield_network.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bzI0fpOmEGY9",
        "colab_type": "text"
      },
      "source": [
        "# How does memory work?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cNSue0AxDStR",
        "colab_type": "text"
      },
      "source": [
        "This is a very exciting field for me: How does human store memory? Is there a way to computationally mimic that mechanism? \n",
        "\n",
        "It is well known in the neuroscience that a specific factoid/memory is not stored in a particular neuron, but stored in the network of neurons and their interconnections. Humans can extract the right memory from partial incorrect information, something that computers can't do yet. \n",
        "\n",
        "In the following I will use a very simple and yet elegant idea from Hopfield (an American scientist, who came up with this idea in 1980's), to demonstrate how we can implement \"memory\" in computer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CW7b38XQEEtu",
        "colab_type": "text"
      },
      "source": [
        "the idea is that we have bunch of neurons, each one can be on or off. And each of them are connected to each other but one neuron is not connected to itself. and the connection is symmetric. \n",
        "\n",
        "Let's do an experiment with a specific example. Suppose we want to remember 5 x 5 pixel image like a T shape. then we will have 25-vector representing neurons' firing. the weight matrix will be 25 x 25, and its diagonal will be zero since no neuron connects to itself. The following is an image of a 8 neuron hopfield network\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ORMJfETGGDC2",
        "colab_type": "text"
      },
      "source": [
        "![hopfield network visualization](https://www.asimovinstitute.org/wp-content/uploads/2016/09/hn.png \"hopfield network visualization\")"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2uCuvVeIDHbH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "t_shape = [1,  1, 1,  1,  1, \n",
        "          -1, -1 ,1, -1, -1,\n",
        "          -1, -1 ,1, -1, -1,\n",
        "          -1, -1, 1, -1, -1,\n",
        "          -1, -1, 1, -1, -1]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pBgWbLk0GsL2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "c_shape = [1, 1, 1, 1, 1, \n",
        "           1,-1,-1,-1,-1,\n",
        "           1,-1,-1,-1,-1,\n",
        "           1,-1,-1,-1,-1,\n",
        "           1, 1, 1, 1, 1]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SPBX2l5RG7PB",
        "colab_type": "text"
      },
      "source": [
        "The idea is that: can our network somehow remembers these two patterns and when given a input that is sorted of like T or sorted of like C, it would give back the original T or C shape? the answer is yes!\n",
        "\n",
        "Now the learning phase is extremely simple, it is just the plain old Hebbian learning rule: **the neurons that fire together, wire together. The neurons that don't fire together inhibit each other**.\n",
        "\n",
        "Mathematically the learning rule is:\n",
        "$$W_{ij}(t+1) = W_{ij}(t) + \\eta s_i s_j$$\n",
        "where $W_{ij}$ is the connection strength from neuron j to neuron i. $\\eta$ is the learning rate. $s_{i}, s_{j}$ are the firing pattern of neuron i, j.\n",
        "\n",
        "Therefore if two neurons both are 1, or both are -1, their connection will strenght in the next time step, if two neurons are of opposite sign, their connection will weaken. And we will do this update rule across the entire network at the same time. in the matrix form we have:\n",
        "$$W(t+1) = W(t) + \\eta (ss^T - I)$$\n",
        "where s is the neuron firing vector, and W is the weight matrix. We minus an identity matrix because if we don't do that $ss^T$ will add $1$'s on the diagonals, which we don't want."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FvnEXvZyKfs-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np # linear algebra library in python"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-OEH3fcjKW7w",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# initializing weight matrix (5,5) of all zeros.\n",
        "W = np.zeros((25,25))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sjGPpQ8hOg3c",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 867
        },
        "outputId": "3d342f6a-42ba-4cc0-c04b-71c51d6e9145"
      },
      "source": [
        "W"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V4cCql43PMUm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# convert python list to numpy array for matrix multiplication\n",
        "# calculations.\n",
        "t_shape = np.asarray(t_shape)\n",
        "c_shape = np.asarray(c_shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ySCt-GBeG5fY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# learning the T pattern in the weight matrix\n",
        "W = W + 0.1 * (t_shape.reshape(25,1) @ t_shape.reshape(1,25) - np.eye(25))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DLNEAfQQPfWI",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "ad7bd414-c8c4-4dd4-b4fa-653122ccb3c5"
      },
      "source": [
        "W"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 0. ,  0.1,  0.1,  0.1,  0.1, -0.1, -0.1,  0.1, -0.1, -0.1, -0.1,\n",
              "        -0.1,  0.1, -0.1, -0.1, -0.1, -0.1,  0.1, -0.1, -0.1, -0.1, -0.1,\n",
              "         0.1, -0.1, -0.1],\n",
              "       [ 0.1,  0. ,  0.1,  0.1,  0.1, -0.1, -0.1,  0.1, -0.1, -0.1, -0.1,\n",
              "        -0.1,  0.1, -0.1, -0.1, -0.1, -0.1,  0.1, -0.1, -0.1, -0.1, -0.1,\n",
              "         0.1, -0.1, -0.1],\n",
              "       [ 0.1,  0.1,  0. ,  0.1,  0.1, -0.1, -0.1,  0.1, -0.1, -0.1, -0.1,\n",
              "        -0.1,  0.1, -0.1, -0.1, -0.1, -0.1,  0.1, -0.1, -0.1, -0.1, -0.1,\n",
              "         0.1, -0.1, -0.1],\n",
              "       [ 0.1,  0.1,  0.1,  0. ,  0.1, -0.1, -0.1,  0.1, -0.1, -0.1, -0.1,\n",
              "        -0.1,  0.1, -0.1, -0.1, -0.1, -0.1,  0.1, -0.1, -0.1, -0.1, -0.1,\n",
              "         0.1, -0.1, -0.1],\n",
              "       [ 0.1,  0.1,  0.1,  0.1,  0. , -0.1, -0.1,  0.1, -0.1, -0.1, -0.1,\n",
              "        -0.1,  0.1, -0.1, -0.1, -0.1, -0.1,  0.1, -0.1, -0.1, -0.1, -0.1,\n",
              "         0.1, -0.1, -0.1],\n",
              "       [-0.1, -0.1, -0.1, -0.1, -0.1,  0. ,  0.1, -0.1,  0.1,  0.1,  0.1,\n",
              "         0.1, -0.1,  0.1,  0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,  0.1,\n",
              "        -0.1,  0.1,  0.1],\n",
              "       [-0.1, -0.1, -0.1, -0.1, -0.1,  0.1,  0. , -0.1,  0.1,  0.1,  0.1,\n",
              "         0.1, -0.1,  0.1,  0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,  0.1,\n",
              "        -0.1,  0.1,  0.1],\n",
              "       [ 0.1,  0.1,  0.1,  0.1,  0.1, -0.1, -0.1,  0. , -0.1, -0.1, -0.1,\n",
              "        -0.1,  0.1, -0.1, -0.1, -0.1, -0.1,  0.1, -0.1, -0.1, -0.1, -0.1,\n",
              "         0.1, -0.1, -0.1],\n",
              "       [-0.1, -0.1, -0.1, -0.1, -0.1,  0.1,  0.1, -0.1,  0. ,  0.1,  0.1,\n",
              "         0.1, -0.1,  0.1,  0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,  0.1,\n",
              "        -0.1,  0.1,  0.1],\n",
              "       [-0.1, -0.1, -0.1, -0.1, -0.1,  0.1,  0.1, -0.1,  0.1,  0. ,  0.1,\n",
              "         0.1, -0.1,  0.1,  0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,  0.1,\n",
              "        -0.1,  0.1,  0.1],\n",
              "       [-0.1, -0.1, -0.1, -0.1, -0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0. ,\n",
              "         0.1, -0.1,  0.1,  0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,  0.1,\n",
              "        -0.1,  0.1,  0.1],\n",
              "       [-0.1, -0.1, -0.1, -0.1, -0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,\n",
              "         0. , -0.1,  0.1,  0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,  0.1,\n",
              "        -0.1,  0.1,  0.1],\n",
              "       [ 0.1,  0.1,  0.1,  0.1,  0.1, -0.1, -0.1,  0.1, -0.1, -0.1, -0.1,\n",
              "        -0.1,  0. , -0.1, -0.1, -0.1, -0.1,  0.1, -0.1, -0.1, -0.1, -0.1,\n",
              "         0.1, -0.1, -0.1],\n",
              "       [-0.1, -0.1, -0.1, -0.1, -0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,\n",
              "         0.1, -0.1,  0. ,  0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,  0.1,\n",
              "        -0.1,  0.1,  0.1],\n",
              "       [-0.1, -0.1, -0.1, -0.1, -0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,\n",
              "         0.1, -0.1,  0.1,  0. ,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,  0.1,\n",
              "        -0.1,  0.1,  0.1],\n",
              "       [-0.1, -0.1, -0.1, -0.1, -0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,\n",
              "         0.1, -0.1,  0.1,  0.1,  0. ,  0.1, -0.1,  0.1,  0.1,  0.1,  0.1,\n",
              "        -0.1,  0.1,  0.1],\n",
              "       [-0.1, -0.1, -0.1, -0.1, -0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,\n",
              "         0.1, -0.1,  0.1,  0.1,  0.1,  0. , -0.1,  0.1,  0.1,  0.1,  0.1,\n",
              "        -0.1,  0.1,  0.1],\n",
              "       [ 0.1,  0.1,  0.1,  0.1,  0.1, -0.1, -0.1,  0.1, -0.1, -0.1, -0.1,\n",
              "        -0.1,  0.1, -0.1, -0.1, -0.1, -0.1,  0. , -0.1, -0.1, -0.1, -0.1,\n",
              "         0.1, -0.1, -0.1],\n",
              "       [-0.1, -0.1, -0.1, -0.1, -0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,\n",
              "         0.1, -0.1,  0.1,  0.1,  0.1,  0.1, -0.1,  0. ,  0.1,  0.1,  0.1,\n",
              "        -0.1,  0.1,  0.1],\n",
              "       [-0.1, -0.1, -0.1, -0.1, -0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,\n",
              "         0.1, -0.1,  0.1,  0.1,  0.1,  0.1, -0.1,  0.1,  0. ,  0.1,  0.1,\n",
              "        -0.1,  0.1,  0.1],\n",
              "       [-0.1, -0.1, -0.1, -0.1, -0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,\n",
              "         0.1, -0.1,  0.1,  0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0. ,  0.1,\n",
              "        -0.1,  0.1,  0.1],\n",
              "       [-0.1, -0.1, -0.1, -0.1, -0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,\n",
              "         0.1, -0.1,  0.1,  0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,  0. ,\n",
              "        -0.1,  0.1,  0.1],\n",
              "       [ 0.1,  0.1,  0.1,  0.1,  0.1, -0.1, -0.1,  0.1, -0.1, -0.1, -0.1,\n",
              "        -0.1,  0.1, -0.1, -0.1, -0.1, -0.1,  0.1, -0.1, -0.1, -0.1, -0.1,\n",
              "         0. , -0.1, -0.1],\n",
              "       [-0.1, -0.1, -0.1, -0.1, -0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,\n",
              "         0.1, -0.1,  0.1,  0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,  0.1,\n",
              "        -0.1,  0. ,  0.1],\n",
              "       [-0.1, -0.1, -0.1, -0.1, -0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,\n",
              "         0.1, -0.1,  0.1,  0.1,  0.1,  0.1, -0.1,  0.1,  0.1,  0.1,  0.1,\n",
              "        -0.1,  0.1,  0. ]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 64
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bg0uLW0mK-ZA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# learning the C pattern in the weight matrix\n",
        "W = W + 0.1 * (c_shape.reshape(25,1) @ c_shape.reshape(1,25) - np.eye(25))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JCXp1cQFLlcQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# recall pattern from corrupted signal\n",
        "corrupted_T = [ 1,  1, 1,  1,  1, \n",
        "               -1, -1 ,1, -1, -1,\n",
        "               -1,  1,-1, -1, -1,\n",
        "               -1,  1,-1, -1, -1,\n",
        "               -1, -1, 1, -1, -1]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tUYD_LKfL8ec",
        "colab_type": "text"
      },
      "source": [
        "The \"recalling mechanism\" involves the neurons change each other's state based on its input until they settle down to a stable pattern:\n",
        "\n",
        "randomly pick a neuron $s_i$, $s_i(t+1) = \\sum_{j\\ne i} W_{ij}s_j(t)$, do this long enough, hopefully we will have a stable network, and hopefully that is the pattern we want to remember."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-RkP-JCHL681",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for i in range(100):\n",
        "  new_vector = W @ corrupted_T\n",
        "  corrupted_T = [1 if x > 0 else -1 for x in new_vector]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KvsWQCaQNrfL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "corrupted_T = np.asarray(corrupted_T)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gGlkKebTN33I",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "fba2b538-8b5f-41e6-b21a-285657e3875b"
      },
      "source": [
        "corrupted_T.reshape((5,5))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 1,  1,  1,  1,  1],\n",
              "       [-1, -1,  1, -1, -1],\n",
              "       [-1, -1,  1, -1, -1],\n",
              "       [-1, -1,  1, -1, -1],\n",
              "       [-1, -1,  1, -1, -1]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mhTRMUTRQW8L",
        "colab_type": "text"
      },
      "source": [
        "Yay! We recovered our T pattern!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hfINL2vQQcoV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "corrupted_C =[1, 1, 1, 1, 1, \n",
        "              1, 1,-1,-1,-1,\n",
        "              1, 1,-1,-1,-1,\n",
        "              1, 1,-1,-1,-1,\n",
        "              1,-1,-1, 1, 1]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FW7J9rlxNRlW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for i in range(100):\n",
        "  new_vector = W @ corrupted_C\n",
        "  corrupted_C = [1 if x > 0 else -1 for x in new_vector]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wlYc1wkrNmLj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "0c534978-7526-4f98-9853-67457c325ab5"
      },
      "source": [
        "corrupted_C = np.asarray(corrupted_C)\n",
        "corrupted_C.reshape((5,5))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 1,  1,  1,  1,  1],\n",
              "       [ 1, -1, -1, -1, -1],\n",
              "       [ 1, -1, -1, -1, -1],\n",
              "       [ 1, -1, -1, -1, -1],\n",
              "       [ 1,  1,  1,  1,  1]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yVwfofzkQy_f",
        "colab_type": "text"
      },
      "source": [
        "we recovered our C pattern as well!!! I think this is super cool, but obvious question remains: how many patterns can we remember? We certainly can't remember 1000 patterns with this mechanism? how much deviation from original pattern can we have but still retrieving the original pattern? (how much corrupted can data be?)\n",
        "\n",
        "[play with hopfield network yourself on this website! the backend algorithm is exactly the same as in this notebook, but with cooler visualization](http://faculty.etsu.edu/knisleyj/neural/neuralnet3.htm)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ihrNJIi_R6-G",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}