session_1/Londonhack-Session-1-Pytorch-Autograd-Optimization-Neural-Networks-Exercise.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pylab inline\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "def set_seed(seed):\n",
    "    \"\"\"\n",
    "    Use this to set ALL the random seeds to a fixed value and take out any randomness from cuda kernels\n",
    "    \"\"\"\n",
    "    random.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    torch.cuda.manual_seed_all(seed)\n",
    "\n",
    "    torch.backends.cudnn.benchmark = False  ##uses the inbuilt cudnn auto-tuner to find the fastest convolution algorithms. -\n",
    "    torch.backends.cudnn.enabled   = False\n",
    "\n",
    "    return True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# London Hack Pytorch Tutorial Session 1\n",
    "----\n",
    "# Pytorch, Automatic Differentiation, Optimization\n",
    "\n",
    "\n",
    "\n",
    "[Pytorch](https://www.pytorch.org) is a (almost) drop-in replacement to numpy functionality but with added automatic differentiation capabilities.  \n",
    "\n",
    "## Awesome [Pytorch](https://www.pytorch.org) functionalities and where to find them:\n",
    "\n",
    "- [torch.Tensor](https://pytorch.org/docs/stable/tensors.html): Fundamental Tensor operations (matmul, sum, mean, transpose, ...)\n",
    "- [torch.nn](https://pytorch.org/docs/stable/nn.html): Specialised functions for implementing (deep) neural networks\n",
    "    - [torch.nn.DataParallel](https://pytorch.org/docs/stable/nn.html#dataparallel-layers-multi-gpu-distributed): Parallel and Distributed Computing\n",
    "    - [Activation Functions](https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity): Sigmoid, Tanh, ReLU, ...\n",
    "    - [Linear Layers](https://pytorch.org/docs/stable/nn.html#linear-layers)\n",
    "    - [Convolutional Layers](https://pytorch.org/docs/stable/nn.html#convolution-layers)\n",
    "    - [Loss Functions](https://pytorch.org/docs/stable/nn.html#loss-functions): MSE-Loss, CrossEntropyLoss, ...\n",
    "- [torch.optim](https://pytorch.org/docs/stable/optim.html): First and Second-order Gradient Descent Optimizers\n",
    "- [torch.autograd](https://pytorch.org/docs/stable/autograd.html): Automatic Differentiation Functionality\n",
    "- [torch.distributions](https://pytorch.org/docs/stable/distributions.html): Probability Distributions\n",
    "- [torch.utils](): Utility functions\n",
    "    - [torch.utils.data](https://pytorch.org/docs/stable/data.html): Contains useful methods to load and handle data\n",
    "- [torchvision](https://pytorch.org/docs/stable/torchvision/index.html): Datasets, Pre-trained Models, Transforms\n",
    "\n",
    "## Task 0: Basic Pytorch Tensor Operations \n",
    "\n",
    "### 0.1: Basic torch.Tensor operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Setting the random seed of torch, numpy and python's random module\n",
    "set_seed(42)\n",
    "\n",
    "#Declare a scalar value\n",
    "a = torch.Tensor(1)\n",
    "print(a, a.item())\n",
    "\n",
    "#Declare a tensor-like another tensor - creates tensor of the same shape\n",
    "b = torch.zeros_like(a)\n",
    "print(b, b.item())\n",
    "\n",
    "#Create a torch.Tensor from a numpy array - doesn't copy memory! :)\n",
    "c = torch.from_numpy(np.array(range(42))) \n",
    "print(c)\n",
    "\n",
    "#Get back the underlying numpy array\n",
    "print(c.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0.2: Basic torch.Tensor operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create a tensor of Gaussian (0-mean, 1-std. dev.) values\n",
    "m = torch.randn(1, 1, 28, 28)\n",
    "print(m.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0.3: Plotting Pytorch Tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(m[0, 0])\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0.3: Using a GPU\n",
    "We can also make use of a GPU (if we have one) by specifying ```.cuda()``` on any Tensors and later Modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = 'cpu'\n",
    "if torch.cuda.device_count() > 0 and torch.cuda.is_available():\n",
    "    print(\"Cuda installed! Running on GPU!\")\n",
    "    device = 'cuda'\n",
    "else:\n",
    "    print(\"No GPU available!\")\n",
    "    \n",
    "#Move tensors around using the .to(device) command\n",
    "tensor_on_device = torch.ones(1).to(device)\n",
    "print(tensor_on_device.device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0.4: Pytorch Autograd with simple functions\n",
    "\n",
    "Pytorch allows us to write pythonic functions and use them in our computational graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = torch.ones(1, requires_grad=True)*np.pi\n",
    "def f(x):\n",
    "    return torch.sin(x)\n",
    "y = f(a)\n",
    "print(a.item(), y.item())\n",
    "\n",
    "# We can explicitly calculate gradients of a function with respect to a parameter or set of parameters:\n",
    "print(\"The derivative is: \", torch.autograd.grad(y, a)[0].item())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0.5: Pytorch Modules\n",
    "\n",
    "We will make a simple parametric parabola as a Pytorch module.  \n",
    "We don't need to write a backward (derivative) for pytorch modules as long as we use pytorch functionality only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ParametricParabola(nn.Module):\n",
    "    def __init__(self, a, b):\n",
    "        super().__init__()\n",
    "        self.a = torch.nn.Parameter(torch.ones(1, requires_grad=True)*a)\n",
    "        self.b = torch.nn.Parameter(torch.ones(1, requires_grad=True)*b)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        return self.a*x*x+self.b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect what the parameters are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = ParametricParabola(0.5, 2)\n",
    "print(f.a)\n",
    "print(f.b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also compute gradients for this example with respect to the parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.zero_grad() #this resets all the gradients within the computational graph.\n",
    "x = torch.ones(1, requires_grad=False)*np.pi\n",
    "y = f(x)\n",
    "y.backward() #call autograd to compute all partial derivatives with respect to all Parameters that require gradients\n",
    "print(f.a.grad, f.b.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0.6 Using Pytorch's Optimization functionality\n",
    "\n",
    "We can use methods from ```torch.optim``` to optimize parameters.  \n",
    "We use this here to find the global minimum of our parabola with gradient descent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seed(42)\n",
    "x = torch.from_numpy(np.array([50])).float()\n",
    "x.requires_grad = True\n",
    "f = ParametricParabola(2., 0.)\n",
    "optimizer = torch.optim.SGD([x], lr=1e-2)\n",
    "for i in range(1000):\n",
    "    optimizer.zero_grad()\n",
    "    value = f(x)\n",
    "    value.backward()\n",
    "    optimizer.step()\n",
    "    if i % 100 == 99:\n",
    "        print(\"Iteration \", i, \" Functional value: \", value.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1: Plotting the Ackley Function using Numpy\n",
    "\n",
    "The [Ackley](https://en.wikipedia.org/wiki/Ackley_function) function is a common test problem for optimization problems with many local minima:\n",
    "\\begin{equation*}\n",
    "f(x, y)= -20 exp [-0.2\\sqrt{0.5(x^2+y^2)}]-exp[0.5\\{cos(2 \\pi x) + cos(2 \\pi y)\\}]+ e +20\n",
    "\\end{equation*}\n",
    "\n",
    "In the following problem tasks we will use pytorch to evaluate this function numerically and study its behavior under common gradient descent techniques."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 1.1: Define the Ackley-Function as a python method \n",
    "- 1.2: Use your previous defined method to plot the function in 3d in the range $(x, y)\\in [-5, 5]$\n",
    "- 1.3: Use your previous defined method Plot the function in 3d in the range $(x, y) \\in [-32, 32]$\n",
    "- 1.4: What do you observe about this function? Why do you think this function is a test problem for optimization?\n",
    "\n",
    "### 1.1: Define the Ackley-Function as a python method "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import cm\n",
    "import numpy as np\n",
    "\n",
    "def ackley(x, y):\n",
    "    # Add your code here\n",
    "    value = None\n",
    "    \n",
    "    return value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2: Use your previous defined method to plot the function in 3d in the range $(x, y)\\in [-5, 5]$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_ackley_3d(x, y):\n",
    "    fig = plt.figure()\n",
    "    ax = fig.gca(projection='3d')\n",
    "\n",
    "    # Make data.\n",
    "    x, y = np.meshgrid(x, y)\n",
    "    \n",
    "    z = ackley(x, y)\n",
    "\n",
    "    # Plot the surface.\n",
    "    surf = ax.plot_surface(x, y, z, cmap=cm.coolwarm,\n",
    "                                   linewidth=0, antialiased=False)\n",
    "\n",
    "    # Add a color bar which maps values to colors.\n",
    "    fig.colorbar(surf, shrink=0.5, aspect=5)\n",
    "\n",
    "    plt.show()\n",
    "\n",
    "x = np.linspace(-5, 5, 100)\n",
    "y = np.linspace(-5, 5, 100)\n",
    "plot_ackley_3d(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3: Use your previous defined method Plot the function in 3d in the range $(x, y) \\in [-32, 32]$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4: What do you observe about this function? Why do you think this function is a test problem for optimization?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2: The Ackley Function in Pytorch, Autograd and Gradient Descent\n",
    "\n",
    "- 2.1: Define the Ackley-Function as a pytorch nn.Module class\n",
    "- 2.2: Try to find the global minimum (solution: 0., 0.) of the Ackley function with Gradient Descent (```torch.optim.SGD```) from a random starting position. Use torch.randn to obtain a sample from a Gaussian.\n",
    "    - Store and plot the corresponding function value as a function of the iteration number (ackley(x) vs. iteration)\n",
    "    - Store and plot the norm of the gradients during optimization.\n",
    "- 2.3: For 10 random starting positions (```torch.randn```) plot the Ackley function in 2D and the optimization trajectories together:\n",
    "    - Do you reach a global minimum?\n",
    "    - What do you observe for the various optimization trajectories.\n",
    "    \n",
    "### 2.1: Define the Ackley-Function as a pytorch nn.Module class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Ackley(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "    \n",
    "    def forward(self, coords):\n",
    "        value = None\n",
    "        # Coords should have a shape of [N_Points x 2] (because of x and y coordinates)\n",
    "        # Add your code here\n",
    "        return value\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2: Try to find the global minimum (solution: 0., 0.) of the Ackley function with Gradient Descent "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3: For 10 random starting positions (```torch.randn```) plot the Ackley function in 2D and the optimization trajectories together:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 3: A Pytorch Module for Regression\n",
    "\n",
    "Pytorch offers a number of modules pre-implemented, many of which we can use for basic and deep neural networks.  \n",
    "Here we will use a simple linear module to perform linear regression of a dataset of points with Gaussian noise.\n",
    "\n",
    "- 3.1: Use the provided pseudo-code to implement linear regression using gradient descent and pytorch's autograd functionality."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.0: The provided linear regression dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seed(42)\n",
    "x = np.linspace(-1, 1, 100)\n",
    "y = x + np.random.normal(0, 0.25, size=(100))\n",
    "\n",
    "x, y = torch.from_numpy(x).float(), torch.from_numpy(y).float()\n",
    "\n",
    "fig, ax = plt.subplots(1, 1, figsize=(12, 12))\n",
    "ax.scatter(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1: Use the provided pseudo-code to implement linear regression using gradient descent and pytorch's autograd functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = nn.Linear(1, 1, bias=True)\n",
    "print([(parameter, parameter.size())for parameter in model.parameters()])\n",
    "\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)\n",
    "criterion = torch.nn.MSELoss()\n",
    "#Pseudo-Code\n",
    "#iterate over the number of epochs (full data passes)\n",
    "    #iterate over all the data points\n",
    "        #reset optimizer gradients\n",
    "        #predict for current data point the y value ~ model(x)\n",
    "        #compute the loss by computing the criterion(y_prediction, y_ground_truth)\n",
    "        #perform backpropagation\n",
    "        #perform 1 optimizer step\n",
    "        \n",
    "#plot the regression line with the original data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Definition of the half-moon dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from sklearn.datasets import make_moons\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "\n",
    "def make_train_test(batch_size, batch_num, test_size, noise=0.05):\n",
    "    \"\"\"\n",
    "    Makes a two-moon train-test dataset with fixed batch size, number and noise level\n",
    "    \"\"\"\n",
    "    X_train, y_train = make_moons(n_samples=batch_size*batch_num, noise=noise)\n",
    "    y_train = y_train.reshape(batch_num, batch_size, 1)\n",
    "    X_train = X_train.reshape(batch_num, batch_size, 2)\n",
    "\n",
    "\n",
    "    X_test, y_test = make_moons(noise=0.1)\n",
    "    y_test = y_test.reshape(test_size, 1)\n",
    "    return X_train, y_train, X_test, y_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Optional Task 4: Backprop-From-Scratch using Numpy\n",
    "\n",
    "In the following exercise you will implement a simple 1-Hidden Layer Neural Network from scratch using ```torch```.\n",
    "We will use a simple toy dataset called the half-moon dataset as a training set.\n",
    "To perform this task we will outline the necessary steps here and provide you with pseudo-code for your implementation.\n",
    "\n",
    "1. Set the random number generator\n",
    "2. Which hyper-parameters will you need to set prior to training?\n",
    "3. Define the size of the input layer (D), the number of hidden layer units (H) and the output layer units (M).\n",
    "    - Suggestion: use a small number of neurons in the hidden layer e.g. H=3\n",
    "4. Define the training and test sets of your half-moon dataset.\n",
    "5. We will use ```sigmoid``` activation functions. Define functions to compute the forward and \"backward\"-pass of the sigmoid. Your function should take in a ```torch.Tensor``` and return a ```torch.Tensor```\n",
    "6. Define the weight tensors of each-layer. Initiallize the Weight-Tensors as ```torch.randn```. You should have two weight tensors W1, W2.\n",
    "7. Within a training loop perform the following operations for the forward pass\n",
    "    - Compute the affine layer transformation $z_1=W_1X$\n",
    "    - Compute the non-linear activation $a_1=\\sigma(z_1)$\n",
    "    - Compute the affine layer transformation $z_2=W_2a_1$\n",
    "    - Compute the non-linear activation $a_2=\\sigma(z_2)$\n",
    "    - Recall the chain-rule:\n",
    "    - Compute the mismatch $\\frac{\\partial{L}}{\\partial{a_2}}=(a_2-y)$\n",
    "    - Compute the gradient of the Loss with respect to the weights of the output layer $\\frac{\\partial{L}}{\\partial{W_2}}=a_1^T*\\frac{\\partial{L}}{\\partial{a_2}}\\frac{\\partial{a_2}}{\\partial{z_2}}$. You will need to use ```torch.transpose``` and ```torch.matmul``` to perform this operation.\n",
    "    - Compute the error on the output of the hidden-layer $\\frac{\\partial{L}}{\\partial{a_1}}$\n",
    "    - Compute the gradient of the loss with respect to the hidden-layer weights $W_1$. This is the same operation as for the output layer.\n",
    "    - Bonus: Compute the sensitivity of the loss with respect to the input $\\frac{\\partial{L}}{\\partial{X}}$\n",
    "    - Perform a gradient descent step on the weights: $W_2^{t+1} = W_2^{t}-\\frac{\\alpha}{N}\\frac{\\partial{L}}{\\partial{W_2}}$. (Hint: the division by $N$ is necessary due to the ```torch.matmul``` operation being an effective summation over all the input examples.\n",
    "    - Compute the training loss as the binary cross entropy $BCE(y, a_2)=\\frac{1}{N}\\sum{y\\cdot log(a_2)+(1-y)\\cdot log(1-a_2)}$\n",
    "8. Perform the above iteration over a number of epochs (full-passes through the training set and use full-batch learning)\n",
    "8. After training, evaluate the performance on the test set by evaluating $y_{pred}=\\sigma(W_2\\sigma(W_1 X))$ and computing the accuracy using ```sklearn.metrics.accuracy_score```.\n",
    "9. Plot the prediction on the training and the test set.\n",
    "10. Bonus: Plot the sensitivity of the loss with respect to each datapoint in the input of the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seed(42)\n",
    "\n",
    "epochs = 1000 #Number of loops through whole dataset\n",
    "\n",
    "batch_size = 1000 #Size of a single batch\n",
    "batch_num = 1 #Use full batch training\n",
    "test_size = 100 #Examples in test set\n",
    "\n",
    "lr = 1.\n",
    "I, H, O = 2, 3, 1 #Define input size (2), Size of Hidden Layer (4), Output size (1)\n",
    "\n",
    "#Use Sklearn to create two-moons + noise\n",
    "X_train, y_train, X_test, y_test = make_train_test(batch_size, batch_num, test_size, noise=0.2)\n",
    "\n",
    "#Define Train Set in Pytorch\n",
    "X = torch.from_numpy(X_train).float()[0] #Convert to torch tensor, single batch\n",
    "y = torch.from_numpy(y_train).float()[0] #Convert to torch tensor, single batch\n",
    "\n",
    "#Define Test Set in Pytorch\n",
    "X_test = torch.from_numpy(X_test).float() #Convert to torch tensor, already single batch\n",
    "y_test = torch.from_numpy(y_test).float() #Convert to torch tensor, already single batch\n",
    "\n",
    "#Define Activation Functions and Derivatives\n",
    "#Sigmoid Activation Function\n",
    "#Derivative of Sigmoid Activation Function\n",
    "\n",
    "#Define Neural Network Parameters\n",
    "#Define the weight matrices\n",
    "\n",
    "#Enter training loop\n",
    "for i in range(epochs):\n",
    "    N = X.size(0) #Number of input examples\n",
    "    #Forward Pass Layer 1\n",
    "    #Affine Layer Transformation z1 = W1*X+b1\n",
    "    #Apply non-linear activation function a1 = sigmoid(z1)\n",
    "    \n",
    "    #Forward Pass Layer 2\n",
    "    #Affine Layer Transformation z2 = W2*a1+b2\n",
    "    #Apply non-linear activation function a2 = sigmoid(z2)\n",
    "\n",
    "    #Backward Pass Layer 2\n",
    "    #Compute Error on Output\n",
    "    #Compute derivative of activation function (Sigmoid)\n",
    "    \n",
    "    #Compute gradient w.r.t. weights in layer 2       \n",
    "    \n",
    "    #Backward Pass Layer 1\n",
    "    #Compute Error on Output of Layer 1\n",
    "    #Compute derivative of activation function (Sigmoid)\n",
    "    \n",
    "    #Compute gradient w.r.t. weights in layer 2\n",
    "\n",
    "    #Sensitivity w.r.t. Input\n",
    "    #Compute gradient w.r.t. input X\n",
    "\n",
    "    #Take a step in gradient direction on layer 1 weights\n",
    "\n",
    "    #Take a step in gradient direction on layer 2 weights\n",
    " \n",
    "    #Compute Average Binary-Crossentropy Loss\n",
    "    if i % 100 == 0:\n",
    "        print(\"Training Loss in epoch \"+str(i)+\": %1.2f\" % train_loss.item())\n",
    "        print(\"Training accuracy in epoch \"+str(i)+\": %1.2f\" % accuracy_score(y, np.where(a2[:, 0].numpy()>0.5, 1, 0)),\"\\n\")\n",
    "        \n",
    "#Do Forward Pass of Test Dataset\n",
    "#Forward Pass Layer 1\n",
    "#Affine Layer Transformation z1 = W1*X\n",
    "#Apply non-linear activation function a1 = sigmoid(z1)\n",
    "\n",
    "#Forward Pass Layer 2\n",
    "#Affine Layer Transformation z2 = W2*a1\n",
    "#Apply non-linear activation function a2 = sigmoid(z2)\n",
    "#Compute Binary-Crossentropy Loss on the Test Set\n",
    "\n",
    "print(\"End of Training -> Testing Phase: \")\n",
    "print(\"Train Loss: %1.2f\" % train_loss.item(), \", Test Loss: %1.2f\" % test_loss.item())\n",
    "print(\"Training accuracy in epoch \"+str(i)+\": %1.2f\" % accuracy_score(y, np.where(a2[:, 0].numpy()>0.5, 1, 0)))\n",
    "print(\"Test accuracy in epoch \"+str(i)+\": %1.2f\" % accuracy_score(y_test, np.where(a_test[:, 0].numpy()>0.5, 1, 0)))      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 5: Training a Neural Network with Pytorch\n",
    "\n",
    "We will now perform the exact same exercise but using Pytorch's autograd functionality. \n",
    "Just as in the first few exercises we can create modules that contain all our \"layers\" and \"activations\".  \n",
    "Pytorch then allows us to automatically compute derivatives from the defined computational graph.  \n",
    "Pytorch essentially remembers all the operations that were performed on a dataset, and if you set ```requires_grad=True``` it will compute a gradient with respect to that parameter once you call backward on it.  \n",
    "Let's see how this works for our 1-hidden layer neural network.\n",
    "\n",
    "Here is the general workflow:\n",
    "\n",
    "- 5.1: Define a SingleHiddenLayer Network as a pytorch module\n",
    "- 5.2: Define the cost function\n",
    "- 5.3: Setup the training function\n",
    "- 5.4: Setup a validation/testing function\n",
    "- 5.5: Create a training/validation/testing split of your data\n",
    "- 5.6: Iterate over your dataset (epoch) and train your network using the train() and validate() methods\n",
    "- 5.7: Make Predictions on the training and test set and plot the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import TensorDataset, DataLoader "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1: Define a SingleHiddenLayer Network as a pytorch module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SingleHiddenLayerNetwork(nn.Module):\n",
    "    def __init__(self, I, H, O):\n",
    "        super(SingleHiddenLayerNetwork, self).__init__()\n",
    "        self.hidden_1 = nn.Linear(I, H, bias=False)\n",
    "        self.output = nn.Linear(H, O, bias=False)\n",
    "        self.activation = nn.Sigmoid()\n",
    "        \n",
    "    def forward(self, X):\n",
    "        z1 = self.hidden_1(X)\n",
    "        a1 = self.activation(z1)\n",
    "        z2 = self.output(a1)\n",
    "        a2 = self.activation(z2)\n",
    "        return a2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2: Define the cost function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bce_loss(y, a2):\n",
    "    return -1/y.size(0)*(y*a2.log()+(1-y)*(1-a2).log()).sum(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3: Setup the training function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, optimizer, data_loader):\n",
    "    model.train()\n",
    "    for X, y in data_loader:\n",
    "        optimizer.zero_grad()\n",
    "        a2 = model(X)\n",
    "        loss = bce_loss(y, a2)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "    \n",
    "    y_pred = np.where(a2[:, 0].detach().numpy()>0.5, 1, 0)\n",
    "    accuracy = accuracy_score(y, y_pred)\n",
    "    return loss, accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4: Setup a validation/testing function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(model, data_loader):\n",
    "    model.eval()\n",
    "    for X, y in data_loader:\n",
    "        with torch.no_grad():\n",
    "            a2 = model(X)\n",
    "            loss = bce_loss(y, a2)\n",
    "    y_pred = np.where(a2[:, 0].numpy()>0.5, 1, 0)\n",
    "    accuracy = accuracy_score(y, y_pred)\n",
    "    return loss, accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.5: Create a training/validation/testing split of your data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Define the size of the input, hidden, and output layers\n",
    "I, H, O = 2, 3, 1\n",
    "\n",
    "#Use Sklearn to create two-moons + noise\n",
    "X_train, y_train, X_test, y_test = make_train_test(batch_size, batch_num, test_size, noise=0.2)\n",
    "\n",
    "#Define Train Set in Pytorch\n",
    "X_train = torch.from_numpy(X_train).float()[0] #Convert to torch tensor, single batch\n",
    "y_train = torch.from_numpy(y_train).float()[0] #Convert to torch tensor, single batch\n",
    "\n",
    "train_dataset = TensorDataset(X_train, y_train)\n",
    "\n",
    "#Define Test Set in Pytorch\n",
    "X_test = torch.from_numpy(X_test).float() #Convert to torch tensor, already single batch\n",
    "y_test = torch.from_numpy(y_test).float() #Convert to torch tensor, already single batch\n",
    "\n",
    "test_dataset = TensorDataset(X_test, y_test)\n",
    "\n",
    "#Use Pytorch's functionality to load data in batches. Here we use full-batch training again.\n",
    "train_loader = DataLoader(train_dataset, batch_size=X_train.size(0), shuffle=True)\n",
    "test_loader = DataLoader(test_dataset, batch_size=X_test.size(0), shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.6: Iterate over your dataset (epoch) and train your network using the train() and validate() methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "network = SingleHiddenLayerNetwork(I, H, O)\n",
    "optim = torch.optim.SGD(network.parameters(), lr=1)\n",
    "for i in range(1000):\n",
    "    train_loss, train_accuracy = train(network, optim, train_loader)\n",
    "    test_loss, test_accuracy = evaluate(network, test_loader)\n",
    "    \n",
    "    if i % 100 == 0:\n",
    "        print(\"Training Loss in epoch \"+str(i)+\": %1.2f\" % train_loss.item())\n",
    "        print(\"Training accuracy in epoch \"+str(i)+\": %1.2f\" % train_accuracy)\n",
    "        print(\"Test Loss in epoch \"+str(i)+\": %1.2f\" % test_loss.item())\n",
    "        print(\"Test accuracy in epoch \"+str(i)+\": %1.2f\" % test_accuracy, \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.7: Make Predictions on the training and test set and plot the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "network.eval()\n",
    "with torch.no_grad():\n",
    "    a_train = network(X_train)\n",
    "    a_test = network(X_test)\n",
    "print(\"Test set accuracy: \", accuracy_score(y_test, np.where(a_test[:, 0].numpy()>0.5, 1, 0)))\n",
    "fig, ax = plt.subplots(1, 2, figsize=(12, 6))\n",
    "ax[0].scatter(X_train[:, 0], X_train[:, 1], c=np.where(a_train[:, 0].numpy()>0.5, 1, 0))\n",
    "ax[1].scatter(X_test[:, 0], X_test[:, 1], c=np.where(a_test[:, 0].numpy()>0.5, 1, 0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks almost the same as our own backprop results!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:dlenv]",
   "language": "python",
   "name": "conda-env-dlenv-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}