t81_558_class_03_5_weights.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_5_weights.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 3: Introduction to TensorFlow**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 3 Material\n",
    "\n",
    "* Part 3.1: Deep Learning and Neural Network Introduction [[Video]](https://www.youtube.com/watch?v=zYnI4iWRmpc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_1_neural_net.ipynb)\n",
    "* Part 3.2: Introduction to Tensorflow and Keras [[Video]](https://www.youtube.com/watch?v=PsE73jk55cE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_2_keras.ipynb)\n",
    "* Part 3.3: Saving and Loading a Keras Neural Network [[Video]](https://www.youtube.com/watch?v=-9QfbGM1qGw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_3_save_load.ipynb)\n",
    "* Part 3.4: Early Stopping in Keras to Prevent Overfitting [[Video]](https://www.youtube.com/watch?v=m1LNunuI2fk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_4_early_stop.ipynb)\n",
    "* **Part 3.5: Extracting Weights and Manual Calculation** [[Video]](https://www.youtube.com/watch?v=7PWgx16kH8s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_5_weights.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3.5: Extracting Keras Weights and Manual Neural Network Calculation\n",
    "\n",
    "In this section we will build a neural network and analyze it down the individual weights.  We will train a simple neural network that learns the XOR function.  It is not hard to simply hand-code the neurons to provide an [XOR function](https://en.wikipedia.org/wiki/Exclusive_or); however, for simplicity, we will allow Keras to train this network for us.  We will just use 100K epochs on the ADAM optimizer.  This is massive overkill, but it gets the result, and our focus here is not on tuning.  The neural network is small.  Two inputs, two hidden neurons, and a single output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cycle #1\n",
      "[[0.33333337]\n",
      " [1.        ]\n",
      " [0.33333337]\n",
      " [0.33333337]]\n",
      "Cycle #2\n",
      "[[6.6666645e-01]\n",
      " [6.6666645e-01]\n",
      " [6.6666645e-01]\n",
      " [2.3841858e-07]]\n",
      "Cycle #3\n",
      "[[0.49999997]\n",
      " [1.        ]\n",
      " [0.49999997]\n",
      " [0.        ]]\n",
      "Cycle #4\n",
      "[[0.49999997]\n",
      " [0.49999997]\n",
      " [0.49999997]\n",
      " [0.49999997]]\n",
      "Cycle #5\n",
      "[[0.49999997]\n",
      " [0.49999997]\n",
      " [0.49999997]\n",
      " [0.49999997]]\n",
      "Cycle #6\n",
      "[[0.3333333]\n",
      " [0.3333333]\n",
      " [1.       ]\n",
      " [0.3333333]]\n",
      "Cycle #7\n",
      "[[6.6666645e-01]\n",
      " [6.6666645e-01]\n",
      " [6.6666645e-01]\n",
      " [2.3841858e-07]]\n",
      "Cycle #8\n",
      "[[0.33333334]\n",
      " [0.33333334]\n",
      " [1.        ]\n",
      " [0.33333334]]\n",
      "Cycle #9\n",
      "[[4.4036580e-07]\n",
      " [1.0000011e+00]\n",
      " [1.0000013e+00]\n",
      " [1.8708772e-06]]\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "import numpy as np\n",
    "\n",
    "# Create a dataset for the XOR function\n",
    "x = np.array([\n",
    "    [0,0],\n",
    "    [1,0],\n",
    "    [0,1],\n",
    "    [1,1]\n",
    "])\n",
    "\n",
    "y = np.array([\n",
    "    0,\n",
    "    1,\n",
    "    1,\n",
    "    0\n",
    "])\n",
    "\n",
    "# Build the network\n",
    "# sgd = optimizers.SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)\n",
    "\n",
    "done = False\n",
    "cycle = 1\n",
    "\n",
    "while not done:\n",
    "    print(\"Cycle #{}\".format(cycle))\n",
    "    cycle+=1\n",
    "    model = Sequential()\n",
    "    model.add(Dense(2, input_dim=2, activation='relu')) \n",
    "    model.add(Dense(1)) \n",
    "    model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "    model.fit(x,y,verbose=0,epochs=10000)\n",
    "\n",
    "    # Predict\n",
    "    pred = model.predict(x)\n",
    "    \n",
    "    # Check if successful.  It takes several runs with this small of a network\n",
    "    done = pred[0]<0.01 and pred[3]<0.01 and pred[1] > 0.9 and pred[2] > 0.9 \n",
    "    print(pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.8708772e-06], dtype=float32)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output above should have two numbers near 0.0 for the first and forth spots (input [[0,0]] and [[1,1]]).  The middle two numbers should be near 1.0 (input [[1,0]] and [[0,1]]).  These numbers are in scientific notation.  Due to random starting weights, it is sometimes necessary to run the above through several cycles to get a good result.\n",
    "\n",
    "Now that the neural network is trained, lets dump the weights.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0B -> L1N0: -1.4406940937042236\n",
      "0B -> L1N1: -1.237845253854175e-06\n",
      "L0N0 -> L1N0 = 1.4406936168670654\n",
      "L0N0 -> L1N1 = 0.809988260269165\n",
      "L0N1 -> L1N0 = 1.4406927824020386\n",
      "L0N1 -> L1N1 = 0.8099883794784546\n",
      "1B -> L2N0: 4.403657953844231e-07\n",
      "L1N0 -> L2N0 = -1.3882228136062622\n",
      "L1N1 -> L2N0 = 1.2345885038375854\n"
     ]
    }
   ],
   "source": [
    "# Dump weights\n",
    "for layerNum, layer in enumerate(model.layers):\n",
    "    weights = layer.get_weights()[0]\n",
    "    biases = layer.get_weights()[1]\n",
    "    \n",
    "    for toNeuronNum, bias in enumerate(biases):\n",
    "        print(f'{layerNum}B -> L{layerNum+1}N{toNeuronNum}: {bias}')\n",
    "    \n",
    "    for fromNeuronNum, wgt in enumerate(weights):\n",
    "        for toNeuronNum, wgt2 in enumerate(wgt):\n",
    "            print(f'L{layerNum}N{fromNeuronNum} -> L{layerNum+1}N{toNeuronNum} = {wgt2}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you rerun this, you probably get different weights.  There are many ways to solve the XOR function.\n",
    "\n",
    "In the next section, we copy/paste the weights from above and recreate the calculations done by the neural network.  Because weights can change with each training, the weights used for the below code came from this:\n",
    "\n",
    "```\n",
    "0B -> L1N0: -1.2913415431976318\n",
    "0B -> L1N1: -3.021530048386012e-08\n",
    "L0N0 -> L1N0 = 1.2913416624069214\n",
    "L0N0 -> L1N1 = 1.1912699937820435\n",
    "L0N1 -> L1N0 = 1.2913411855697632\n",
    "L0N1 -> L1N1 = 1.1912697553634644\n",
    "1B -> L2N0: 7.626241297587034e-36\n",
    "L1N0 -> L2N0 = -1.548777461051941\n",
    "L1N1 -> L2N0 = 0.8394404649734497\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0\n",
      "1.2\n",
      "0\n",
      "1.2\n",
      "0.96\n",
      "0.96\n"
     ]
    }
   ],
   "source": [
    "input0 = 0\n",
    "input1 = 1\n",
    "\n",
    "hidden0Sum = (input0*1.3)+(input1*1.3)+(-1.3)\n",
    "hidden1Sum = (input0*1.2)+(input1*1.2)+(0)\n",
    "\n",
    "print(hidden0Sum) # 0\n",
    "print(hidden1Sum) # 1.2\n",
    "\n",
    "hidden0 = max(0,hidden0Sum)\n",
    "hidden1 = max(0,hidden1Sum)\n",
    "\n",
    "print(hidden0) # 0\n",
    "print(hidden1) # 1.2\n",
    "\n",
    "outputSum = (hidden0*-1.6)+(hidden1*0.8)+(0)\n",
    "print(outputSum) # 0.96\n",
    "\n",
    "output = max(0,outputSum)\n",
    "\n",
    "print(output) # 0.96"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "rga"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}