Adversarial_Data_Programming.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Adversarial_Data_Programming.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "mount_file_id": "1PfRQX1nCH4QH0B5Uff_Qzz7lUwjDCM04",
      "authorship_tag": "ABX9TyOTrHNJvVfq0/ZHPhdA+J5I",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ArghyaPal/Adversarial_Data_Programming/blob/master/Adversarial_Data_Programming.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ZEmJOpFtYhN"
      },
      "source": [
        "# Adversarial Data Programming\n",
        "Paucity of large curated hand-labeled training data forms a major bottleneck in the deployment of machine learning models in computer vision and other fields. Recent work (Data Programming) has shown how distant supervision signals in the form of labeling functions can be used to obtain labels for given data in near-constant time. In this work, we present Adversarial Data Programming (ADP), which presents an adversarial methodology to generate data as well as a curated aggregated label, given a set of weak labeling functions. This notebook, we'll show the validation on MNIST dataset. We conducted extensive experiments to study its usefulness, as well as showed how the proposed ADP framework can be used for transfer learning as well as multi-task learning, where data from two domains are generated simultaneously using the framework along with the label information. Our future work will involve understanding the theoretical implications of this new framework from a game-theoretic perspective, as well as explore the performance of the method on more complex datasets.\n",
        "\n",
        "**TL;DR:**<br>\n",
        "Learning the parameters of Joint Distribution, i.e. P(Χ, y) and Χ: image, y: label, using Distant Supervision Signals. In this work, we seek to learn the parameters of the Joint Distribution P(Χ, LFB_(L, Theta)(X)). The LFB_(L, Theta) (.) acts as a proxy to the real label y. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aP3xDdKxMDnM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "77b8057b-1e9d-41f2-a34c-7564fb6814db"
      },
      "source": [
        "# Please run it once during a runtime\n",
        "!pip install kmeans-pytorch"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: kmeans-pytorch in /usr/local/lib/python3.7/dist-packages (0.3)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z5HmixuLKuwt"
      },
      "source": [
        "import os\n",
        "import torch\n",
        "import torchvision\n",
        "import torch.nn as nn\n",
        "from torchvision import transforms\n",
        "from torchvision.utils import save_image\n",
        "from torch.autograd import Variable\n",
        "import matplotlib.pyplot as plt\n",
        "import pylab\n",
        "import numpy as np\n",
        "from kmeans_pytorch import kmeans, kmeans_predict"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hyyWXMNpM0w8"
      },
      "source": [
        "# Decide which device we want to run on\n",
        "ngpu = 1\n",
        "device = torch.device(\"cuda:0\" if (torch.cuda.is_available() and ngpu > 0) else \"cpu\")"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lK2xHmM4K1mQ"
      },
      "source": [
        "'''\n",
        "Hyperparameters used in this projects\n",
        "'''\n",
        "# The noise dimension for the Generator, i.e. the z dimension, sampled from a Gaussian Distribution\n",
        "latent_size = 64\n",
        "# The network layer dimension of Generator and Discriminator\n",
        "hidden_size = 256\n",
        "# The generated image size/image size. MNIST size 28 X 28 = 784\n",
        "image_size = 784\n",
        "# Number of epochs\n",
        "num_epochs = 300\n",
        "# A mini-batch size\n",
        "batch_size = 40\n",
        "# Adam learning rate\n",
        "lr = 0.0001\n",
        "\n",
        "# Number of classes\n",
        "nclass = 10          # For MNIST dataset, please change it for other dataset"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pjsGaQxmHMui"
      },
      "source": [
        "### Please mention the number of Labeleing Functions here"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kD1x0b2aHEtA"
      },
      "source": [
        "theta = 2"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2gZ0G-2oH0BK"
      },
      "source": [
        "### Dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nPedqD1nK4vC"
      },
      "source": [
        "sample_dir = 'samples'\n",
        "save_dir = 'save'\n",
        "\n",
        "# Create a directory if not exists\n",
        "if not os.path.exists(sample_dir):\n",
        "    os.makedirs(sample_dir)\n",
        "\n",
        "if not os.path.exists(save_dir):\n",
        "    os.makedirs(save_dir)\n",
        "\n",
        "# Image processing\n",
        "transform = transforms.Compose([\n",
        "                transforms.ToTensor(),\n",
        "                transforms.Normalize(mean=(0.5),   # 3 for RGB channels\n",
        "                                     std=(0.5))])\n",
        "\n",
        "# MNIST dataset\n",
        "mnist = torchvision.datasets.MNIST(root='./data/',\n",
        "                                   train=True,\n",
        "                                   transform=transform,\n",
        "                                   download=True)\n",
        "\n",
        "# Data loader\n",
        "data_loader = torch.utils.data.DataLoader(dataset=mnist,\n",
        "                                          batch_size=batch_size, \n",
        "                                          shuffle=True)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ievAvjo4B1fm"
      },
      "source": [
        "\n",
        "import torch\n",
        "import numpy as np\n",
        "\n",
        "from math import pi\n",
        "from scipy.special import logsumexp\n",
        "\n",
        "\n",
        "class GaussianMixture(torch.nn.Module):\n",
        "    \"\"\"\n",
        "    Fits a mixture of k=1,..,K Gaussians to the input data (K is supplied via n_components). Input tensors are expected to be flat with dimensions (n: number of samples, d: number of features).\n",
        "    The model then extends them to (n, 1, d).\n",
        "    The model parametrization (mu, sigma) is stored as (1, k, d), and probabilities are shaped (n, k, 1) if they relate to an individual sample, or (1, k, 1) if they assign membership probabilities to one of the mixture components.\n",
        "    \"\"\"\n",
        "    def __init__(self, n_components, n_features, mu_init=None, var_init=None, eps=1.e-6):\n",
        "        \"\"\"\n",
        "        Initializes the model and brings all tensors into their required shape.\n",
        "        The class expects data to be fed as a flat tensor in (n, d).\n",
        "        The class owns:\n",
        "            x:              torch.Tensor (n, 1, d)\n",
        "            mu:             torch.Tensor (1, k, d)\n",
        "            var:            torch.Tensor (1, k, d)\n",
        "            pi:             torch.Tensor (1, k, 1)\n",
        "            eps:            float\n",
        "            n_components:   int\n",
        "            n_features:     int\n",
        "            log_likelihood: float\n",
        "        args:\n",
        "            n_components:   int\n",
        "            n_features:     int\n",
        "        options:\n",
        "            mu_init:        torch.Tensor (1, k, d)\n",
        "            var_init:       torch.Tensor (1, k, d)\n",
        "            eps:            float\n",
        "        \"\"\"\n",
        "        super(GaussianMixture, self).__init__()\n",
        "\n",
        "        self.n_components = n_components\n",
        "        self.n_features = n_features\n",
        "\n",
        "        self.mu_init = mu_init\n",
        "        self.var_init = var_init\n",
        "        self.eps = eps\n",
        "\n",
        "        self.log_likelihood = -np.inf\n",
        "\n",
        "        self._init_params()\n",
        "\n",
        "\n",
        "    def _init_params(self):\n",
        "        if self.mu_init is not None:\n",
        "            assert self.mu_init.size() == (1, self.n_components, self.n_features), \"Input mu_init does not have required tensor dimensions (1, %i, %i)\" % (self.n_components, self.n_features)\n",
        "            # (1, k, d)\n",
        "            self.mu = torch.nn.Parameter(self.mu_init, requires_grad=False)\n",
        "        else:\n",
        "            self.mu = torch.nn.Parameter(torch.randn(1, self.n_components, self.n_features), requires_grad=False)\n",
        "\n",
        "        if self.var_init is not None:\n",
        "            assert self.var_init.size() == (1, self.n_components, self.n_features), \"Input var_init does not have required tensor dimensions (1, %i, %i)\" % (self.n_components, self.n_features)\n",
        "            # (1, k, d)\n",
        "            self.var = torch.nn.Parameter(self.var_init, requires_grad=False)\n",
        "        else:\n",
        "            self.var = torch.nn.Parameter(torch.ones(1, self.n_components, self.n_features), requires_grad=False)\n",
        "\n",
        "        # (1, k, 1)\n",
        "        self.pi = torch.nn.Parameter(torch.Tensor(1, self.n_components, 1), requires_grad=False).fill_(1./self.n_components)\n",
        "\n",
        "        self.params_fitted = False\n",
        "\n",
        "\n",
        "    def check_size(self, x):\n",
        "        if len(x.size()) == 2:\n",
        "            # (n, d) --> (n, 1, d)\n",
        "            x = x.unsqueeze(1)\n",
        "\n",
        "        return x\n",
        "\n",
        "\n",
        "    def bic(self, x):\n",
        "        \"\"\"\n",
        "        Bayesian information criterion for a batch of samples.\n",
        "        args:\n",
        "            x:      torch.Tensor (n, d) or (n, 1, d)\n",
        "        returns:\n",
        "            bic:    float\n",
        "        \"\"\"\n",
        "        x = self.check_size(x)\n",
        "        n = x.shape[0]\n",
        "\n",
        "        # Free parameters for covariance, means and mixture components\n",
        "        free_params = self.n_features * self.n_components + self.n_features + self.n_components - 1\n",
        "\n",
        "        bic = -2. * self.__score(x, sum_data=False).mean() * n + free_params * np.log(n)\n",
        "\n",
        "        return bic\n",
        "\n",
        "\n",
        "    def fit(self, x, delta=1e-3, n_iter=100, warm_start=False):\n",
        "        \"\"\"\n",
        "        Fits model to the data.\n",
        "        args:\n",
        "            x:          torch.Tensor (n, d) or (n, k, d)\n",
        "        options:\n",
        "            delta:      float\n",
        "            n_iter:     int\n",
        "            warm_start: bool\n",
        "        \"\"\"\n",
        "        if not warm_start and self.params_fitted:\n",
        "            self._init_params()\n",
        "\n",
        "        x = self.check_size(x)\n",
        "\n",
        "        i = 0\n",
        "        j = np.inf\n",
        "\n",
        "        while (i <= n_iter) and (j >= delta):\n",
        "\n",
        "            log_likelihood_old = self.log_likelihood\n",
        "            mu_old = self.mu\n",
        "            var_old = self.var\n",
        "\n",
        "            self.__em(x)\n",
        "            self.log_likelihood = self.__score(x)\n",
        "\n",
        "            if (self.log_likelihood.abs() == float(\"Inf\")) or (self.log_likelihood == float(\"nan\")):\n",
        "                # When the log-likelihood assumes inane values, reinitialize model\n",
        "                self.__init__(self.n_components,\n",
        "                    self.n_features,\n",
        "                    mu_init=self.mu_init,\n",
        "                    var_init=self.var_init,\n",
        "                    eps=self.eps)\n",
        "\n",
        "            i += 1\n",
        "            j = self.log_likelihood - log_likelihood_old\n",
        "\n",
        "            if j <= delta:\n",
        "                # When score decreases, revert to old parameters\n",
        "                self.__update_mu(mu_old)\n",
        "                self.__update_var(var_old)\n",
        "\n",
        "        self.params_fitted = True\n",
        "\n",
        "\n",
        "    def predict(self, x, probs=False):\n",
        "        \"\"\"\n",
        "        Assigns input data to one of the mixture components by evaluating the likelihood under each.\n",
        "        If probs=True returns normalized probabilities of class membership.\n",
        "        args:\n",
        "            x:          torch.Tensor (n, d) or (n, 1, d)\n",
        "            probs:      bool\n",
        "        returns:\n",
        "            p_k:        torch.Tensor (n, k)\n",
        "            (or)\n",
        "            y:          torch.LongTensor (n)\n",
        "        \"\"\"\n",
        "        x = self.check_size(x)\n",
        "\n",
        "        weighted_log_prob = self._estimate_log_prob(x) + torch.log(self.pi)\n",
        "\n",
        "        if probs:\n",
        "            p_k = torch.exp(weighted_log_prob)\n",
        "            return torch.squeeze(p_k / (p_k.sum(1, keepdim=True)))\n",
        "        else:\n",
        "            return torch.squeeze(torch.max(weighted_log_prob, 1)[1].type(torch.LongTensor))\n",
        "\n",
        "\n",
        "    def predict_proba(self, x):\n",
        "        \"\"\"\n",
        "        Returns normalized probabilities of class membership.\n",
        "        args:\n",
        "            x:          torch.Tensor (n, d) or (n, 1, d)\n",
        "        returns:\n",
        "            y:          torch.LongTensor (n)\n",
        "        \"\"\"\n",
        "        return self.predict(x, probs=True)\n",
        "\n",
        "\n",
        "    def score_samples(self, x):\n",
        "        \"\"\"\n",
        "        Computes log-likelihood of samples under the current model.\n",
        "        args:\n",
        "            x:          torch.Tensor (n, d) or (n, 1, d)\n",
        "        returns:\n",
        "            score:      torch.LongTensor (n)\n",
        "        \"\"\"\n",
        "        x = self.check_size(x)\n",
        "\n",
        "        score = self.__score(x, sum_data=False)\n",
        "        return score\n",
        "\n",
        "\n",
        "    def _estimate_log_prob(self, x):\n",
        "        \"\"\"\n",
        "        Returns a tensor with dimensions (n, k, 1), which indicates the log-likelihood that samples belong to the k-th Gaussian.\n",
        "        args:\n",
        "            x:            torch.Tensor (n, d) or (n, 1, d)\n",
        "        returns:\n",
        "            log_prob:     torch.Tensor (n, k, 1)\n",
        "        \"\"\"\n",
        "        x = self.check_size(x)\n",
        "\n",
        "        mu = self.mu\n",
        "        prec = torch.rsqrt(self.var)\n",
        "\n",
        "        log_p = torch.sum((mu * mu + x * x - 2 * x * mu) * (prec ** 2), dim=2, keepdim=True)\n",
        "        log_det = torch.sum(torch.log(prec), dim=2, keepdim=True)\n",
        "\n",
        "        return -.5 * (self.n_features * np.log(2. * pi) + log_p) + log_det\n",
        "\n",
        "\n",
        "    def _e_step(self, x):\n",
        "        \"\"\"\n",
        "        Computes log-responses that indicate the (logarithmic) posterior belief (sometimes called responsibilities) that a data point was generated by one of the k mixture components.\n",
        "        Also returns the mean of the mean of the logarithms of the probabilities (as is done in sklearn).\n",
        "        This is the so-called expectation step of the EM-algorithm.\n",
        "        args:\n",
        "            x:              torch.Tensor (n,d) or (n, 1, d)\n",
        "        returns:\n",
        "            log_prob_norm:  torch.Tensor (1)\n",
        "            log_resp:       torch.Tensor (n, k, 1)\n",
        "        \"\"\"\n",
        "        x = self.check_size(x)\n",
        "\n",
        "        weighted_log_prob = self._estimate_log_prob(x) + torch.log(self.pi)\n",
        "\n",
        "        log_prob_norm = torch.logsumexp(weighted_log_prob, dim=1, keepdim=True)\n",
        "        log_resp = weighted_log_prob - log_prob_norm\n",
        "\n",
        "        return torch.mean(log_prob_norm), log_resp\n",
        "\n",
        "\n",
        "    def _m_step(self, x, log_resp):\n",
        "        \"\"\"\n",
        "        From the log-probabilities, computes new parameters pi, mu, var (that maximize the log-likelihood). This is the maximization step of the EM-algorithm.\n",
        "        args:\n",
        "            x:          torch.Tensor (n, d) or (n, 1, d)\n",
        "            log_resp:   torch.Tensor (n, k, 1)\n",
        "        returns:\n",
        "            pi:         torch.Tensor (1, k, 1)\n",
        "            mu:         torch.Tensor (1, k, d)\n",
        "            var:        torch.Tensor (1, k, d)\n",
        "        \"\"\"\n",
        "        x = self.check_size(x)\n",
        "\n",
        "        resp = torch.exp(log_resp)\n",
        "\n",
        "        pi = torch.sum(resp, dim=0, keepdim=True) + self.eps\n",
        "        mu = torch.sum(resp * x, dim=0, keepdim=True) / pi\n",
        "\n",
        "        x2 = (resp * x * x).sum(0, keepdim=True) / pi\n",
        "        mu2 = mu * mu\n",
        "        xmu = (resp * mu * x).sum(0, keepdim=True) / pi\n",
        "        var = x2 - 2 * xmu + mu2 + self.eps\n",
        "\n",
        "        pi = pi / x.shape[0]\n",
        "\n",
        "        return pi, mu, var\n",
        "\n",
        "\n",
        "    def __em(self, x):\n",
        "        \"\"\"\n",
        "        Performs one iteration of the expectation-maximization algorithm by calling the respective subroutines.\n",
        "        args:\n",
        "            x:          torch.Tensor (n, 1, d)\n",
        "        \"\"\"\n",
        "        _, log_resp = self._e_step(x)\n",
        "        pi, mu, var = self._m_step(x, log_resp)\n",
        "\n",
        "        self.__update_pi(pi)\n",
        "        self.__update_mu(mu)\n",
        "        self.__update_var(var)\n",
        "\n",
        "\n",
        "    def __score(self, x, sum_data=True):\n",
        "        \"\"\"\n",
        "        Computes the log-likelihood of the data under the model.\n",
        "        args:\n",
        "            x:                  torch.Tensor (n, 1, d)\n",
        "            sum_data:           bool\n",
        "        returns:\n",
        "            score:              torch.Tensor (1)\n",
        "            (or)\n",
        "            per_sample_score:   torch.Tensor (n)\n",
        "\n",
        "        \"\"\"\n",
        "        weighted_log_prob = self._estimate_log_prob(x) + torch.log(self.pi)\n",
        "        per_sample_score = torch.logsumexp(weighted_log_prob, dim=1)\n",
        "\n",
        "        if sum_data:\n",
        "            return per_sample_score.sum()\n",
        "        else:\n",
        "            return torch.squeeze(per_sample_score)\n",
        "\n",
        "\n",
        "    def __update_mu(self, mu):\n",
        "        \"\"\"\n",
        "        Updates mean to the provided value.\n",
        "        args:\n",
        "            mu:         torch.FloatTensor\n",
        "        \"\"\"\n",
        "\n",
        "        assert mu.size() in [(self.n_components, self.n_features), (1, self.n_components, self.n_features)], \"Input mu does not have required tensor dimensions (%i, %i) or (1, %i, %i)\" % (self.n_components, self.n_features, self.n_components, self.n_features)\n",
        "\n",
        "        if mu.size() == (self.n_components, self.n_features):\n",
        "            self.mu = mu.unsqueeze(0)\n",
        "        elif mu.size() == (1, self.n_components, self.n_features):\n",
        "            self.mu.data = mu\n",
        "\n",
        "\n",
        "    def __update_var(self, var):\n",
        "        \"\"\"\n",
        "        Updates variance to the provided value.\n",
        "        args:\n",
        "            var:        torch.FloatTensor\n",
        "        \"\"\"\n",
        "\n",
        "        assert var.size() in [(self.n_components, self.n_features), (1, self.n_components, self.n_features)], \"Input var does not have required tensor dimensions (%i, %i) or (1, %i, %i)\" % (self.n_components, self.n_features, self.n_components, self.n_features)\n",
        "\n",
        "        if var.size() == (self.n_components, self.n_features):\n",
        "            self.var = var.unsqueeze(0)\n",
        "        elif var.size() == (1, self.n_components, self.n_features):\n",
        "            self.var.data = var\n",
        "\n",
        "\n",
        "    def __update_pi(self, pi):\n",
        "        \"\"\"\n",
        "        Updates pi to the provided value.\n",
        "        args:\n",
        "            pi:         torch.FloatTensor\n",
        "        \"\"\"\n",
        "\n",
        "        assert pi.size() in [(1, self.n_components, 1)], \"Input pi does not have required tensor dimensions (%i, %i, %i)\" % (1, self.n_components, 1)\n",
        "\n",
        "        self.pi.data = pi"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cpE1mtRn__m0"
      },
      "source": [
        "### ADP Labeling Function Block $LFB(y|X, \\Theta)$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d0OYu7dJ0GKD"
      },
      "source": [
        "    \"\"\"\n",
        "    The labeling functions infer label of an image independently \n",
        "    (i.e. independent decision assumption) and the parameter gives relative \n",
        "    weight to each of the labeling functions based on their correctness of \n",
        "    inferred label for true class y.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    X\n",
        "        Batch of images, torch.Tensor\n",
        "    theta\n",
        "        Batch of relative accuracies are weights given to labeling functions \n",
        "        based on whether their outputs agree with true class label y of an \n",
        "        image X, torch.Tensor\n",
        "    Returns\n",
        "    -------\n",
        "    theta * y\n",
        "        Batch of approximated labels (in this formulation), torch.Tensor\n",
        "    \"\"\"\n",
        "def lf1(fake_images, theta_1): \n",
        "  # fake_images is a batch of generated images from the G_image\n",
        "  cluster_centers = torch.load('cluster_centers.pt') # please set the path \n",
        "                                                                            \n",
        "  # we use k-means (unsupervised) clustering as one labeling function\n",
        "  y1 = kmeans_predict(fake_images, cluster_centers, 'euclidean', device=device)\n",
        "  y1 = y1.view(-1,1).cuda()\n",
        "  theta_1 = torch.reshape(theta_1, (40, 1))\n",
        "  return torch.mul(theta_1,y1)\n",
        "\n",
        "def lf2(fake_images, theta_2):\n",
        "  model = torch.load('gmm.pt')\n",
        "  y2 = model.predict(fake_images)\n",
        "  y2 = y2.view(-1,1).cuda()\n",
        "  theta_2 = torch.reshape(theta_2, (40, 1))\n",
        "  return torch.mul(theta_2,y2)"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rsci7azICCBv"
      },
      "source": [
        "### ADP Generator $G(X, \\Theta|z)$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Tyl-RaOiLM1u"
      },
      "source": [
        "class Generator(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Generator, self).__init__()\n",
        "        self.G_common = nn.Sequential(\n",
        "                              nn.Linear(latent_size, hidden_size),\n",
        "                              nn.ReLU(),\n",
        "                              nn.Linear(hidden_size, hidden_size),\n",
        "                              nn.ReLU())\n",
        "        self.G_image = nn.Sequential(\n",
        "                              nn.Linear(hidden_size, image_size),\n",
        "                              nn.Tanh())\n",
        "        self.G_parameter = nn.Sequential(\n",
        "                              nn.Linear(hidden_size, theta),\n",
        "                              # We use softmax to make the theta to be in the \n",
        "                              # range [0,1]\n",
        "                              nn.Softmax(dim=1))\n",
        "    \"\"\"\n",
        "    Parameters\n",
        "    ----------\n",
        "    z\n",
        "        Batch of noise vector, torch.Tensor\n",
        "    Returns\n",
        "    -------\n",
        "    X\n",
        "        Batch of images, torch.Tensor\n",
        "    theta\n",
        "        Batch of a Relative accuracies, torch.Tensor\n",
        "    \"\"\"\n",
        "    def forward(self, input):\n",
        "        common = self.G_common(input)\n",
        "        image = self.G_image(common)\n",
        "        theta = self.G_parameter(common)\n",
        "        return image, theta"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QbZng0nN1HHe"
      },
      "source": [
        "### ADP Discriminator D(X, y)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "58NhqczSLKTw"
      },
      "source": [
        "  '''\n",
        "  The discriminator class takes a batch of either real or generated image and \n",
        "  inferred label from Labeling Functions Block LFB(.) pairs as input.\n",
        "  And, maps that to a probability score to estimate the aforementioned \n",
        "  likelihood of the image-label pair as real labeled-image\n",
        "  '''\n",
        "class Discriminator(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Discriminator, self).__init__()\n",
        "        '''\n",
        "        The D_image takes a batch of images\n",
        "        '''\n",
        "        self.D_image = nn.Sequential(\n",
        "                              nn.Linear(image_size, hidden_size),\n",
        "                              nn.LeakyReLU(0.2))\n",
        "        '''\n",
        "        The D_label takes a batch of labels\n",
        "        '''\n",
        "        self.D_label = nn.Sequential(\n",
        "                              nn.Linear(10, hidden_size),\n",
        "                              nn.LeakyReLU(0.2))\n",
        "        '''\n",
        "        The D_common adds the D_image and D_label branches and provides \n",
        "        '''\n",
        "        self.D_common = nn.Sequential(\n",
        "                              nn.Linear(hidden_size*2, hidden_size),\n",
        "                              nn.LeakyReLU(0.2),\n",
        "                              nn.Linear(hidden_size, 1),\n",
        "                              # This will ensure the output to be in the range of [0,1]\n",
        "                              nn.Sigmoid())\n",
        "    \"\"\"Estimates the probability score as a likelihood of the input image-label pair\n",
        "    as real image-label pair.\n",
        "    This function provides a batch of probability scores, i.e. P(X, y), stored as a \n",
        "    vector P. We will use that vector to give loss signal to the Discriminator D(.) \n",
        "    and Generator G(.).\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    X\n",
        "        Batch of images, torch.Tensor\n",
        "    y\n",
        "        Batch of labels, torch.Tensor\n",
        "    Returns\n",
        "    -------\n",
        "    result\n",
        "        Batch of probabilities, torch.Tensor\n",
        "    \"\"\"\n",
        "    def forward(self, image, label):\n",
        "        # image is the batch of images given to the D_image branch\n",
        "        image = self.D_image(image)\n",
        "        # label is the corresponding labels given to the D_label branch\n",
        "        label = self.D_label(label)\n",
        "        # common is the branch that concates D_image and D_label\n",
        "        common = torch.cat([image, label], 1)\n",
        "        # result is the batch of probability score\n",
        "        result = self.D_common(common)\n",
        "        return result"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YlfNq55ULRuN"
      },
      "source": [
        "# Device setting\n",
        "G = Generator().cuda()\n",
        "D = Discriminator().cuda()\n",
        "\n",
        "# Binary cross entropy loss and optimizer\n",
        "criterion = nn.BCELoss()\n",
        "d_optimizer = torch.optim.Adam(D.parameters(), lr=lr)\n",
        "g_optimizer = torch.optim.Adam(G.parameters(), lr=lr)\n",
        "\n",
        "def denorm(x):\n",
        "    out = (x + 1) / 2\n",
        "    return out.clamp(0, 1)\n",
        "\n",
        "def reset_grad():\n",
        "    d_optimizer.zero_grad()\n",
        "    g_optimizer.zero_grad()"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kFY03OEhLWKc"
      },
      "source": [
        "# Statistics to be saved\n",
        "d_losses = np.zeros(num_epochs)\n",
        "g_losses = np.zeros(num_epochs)\n",
        "real_scores = np.zeros(num_epochs)\n",
        "fake_scores = np.zeros(num_epochs)"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n9TM5beFLhi4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "be5d0b01-ac03-4229-bdfa-8bdf158b54b0"
      },
      "source": [
        "# We will start Adversarial Data Programming training\n",
        "total_step = len(data_loader)\n",
        "\n",
        "# To change num_epochhs please visit the hyperparameters section\n",
        "for epoch in range(num_epochs):\n",
        "  # We will be training discriminator one time and then the generator one time\n",
        "  # using the for loop\n",
        "    for i, (images, original_labels) in enumerate(data_loader):\n",
        "\n",
        "        # Reading real MNIST images\n",
        "        images = images.view(batch_size, -1).cuda()\n",
        "        images = Variable(images)\n",
        "        # Reading real MNIST labeles of the corresponding images\n",
        "        original_labels = torch.nn.functional.one_hot(original_labels, num_classes=10)\n",
        "        # We are making it in one-hot encoding\n",
        "        original_labels = original_labels.view(batch_size, -1).cuda()\n",
        "        original_labels = Variable(original_labels).float()\n",
        "        \n",
        "\n",
        "        # Labels for the discriminator. Real Labels are 1 and Fake Labels are 0\n",
        "        real_labels = torch.ones(batch_size, 1).cuda()\n",
        "        real_labels = Variable(real_labels)\n",
        "        fake_labels = torch.zeros(batch_size, 1).cuda()\n",
        "        fake_labels = Variable(fake_labels)\n",
        "\n",
        "        #######################################################################\n",
        "        '''\n",
        "        Below line of codes will train the discriminator.\n",
        "        The discriminator will take: (i) a batch of images; and (ii) a batch of labels\n",
        "        The object, i.e. outputs = D(images, original_labels), of the Discriminator class\n",
        "        We compute BCE_Loss using real images where:\n",
        "        BCE_Loss(x, y): - y * log(D(x)) - (1-y) * log(1 - D(x))\n",
        "        Second term of the loss is always zero since real_labels == 1\n",
        "        '''\n",
        "        outputs = D(images, original_labels)\n",
        "        d_loss_real = criterion(outputs, real_labels)\n",
        "        real_score = outputs\n",
        "        #######################################################################\n",
        "\n",
        "        #######################################################################\n",
        "        '''\n",
        "        We sample z from the Isotropic Gaussian distribution\n",
        "        Compute BCELoss using fake images\n",
        "        First term of the loss is always zero since fake_labels == 0\n",
        "        '''\n",
        "        z = torch.randn(batch_size, latent_size).cuda()\n",
        "        z = Variable(z)\n",
        "        # We put that in the G_common\n",
        "        fake_images, theta = G(z)\n",
        "\n",
        "        # generated labels from labeling function (here, we are showing one labeling function)\n",
        "        gen_labels_1 = lf1(fake_images, theta[:,0])\n",
        "        #gen_labels_1 = torch.nn.functional.one_hot(gen_labels_1.to(torch.int64), num_classes=nclass).float()\n",
        "        gen_labels_2 = lf2(fake_images, theta[:,1])\n",
        "        #gen_labels_2 = torch.nn.functional.one_hot(gen_labels_2.to(torch.int64), num_classes=nclass).float()\n",
        "    \n",
        "\n",
        "        gen_labels = gen_labels_1 + gen_labels_2\n",
        "        gen_labels = torch.nn.functional.one_hot(gen_labels.to(torch.int64), num_classes=nclass).float()\n",
        "        \n",
        "        gen_labels = gen_labels.view(batch_size, 10)\n",
        "        \n",
        "        # \n",
        "        outputs = D(fake_images, gen_labels)\n",
        "        d_loss_fake = criterion(outputs, fake_labels)\n",
        "        fake_score = outputs\n",
        "        \n",
        "        # Backprop and optimize\n",
        "        # If D is trained so well, then don't update\n",
        "        d_loss = d_loss_real + d_loss_fake\n",
        "        reset_grad()\n",
        "        d_loss.backward()\n",
        "        d_optimizer.step()\n",
        "        #######################################################################\n",
        "\n",
        "        '''\n",
        "        Train generator \n",
        "        '''\n",
        "\n",
        "        # Compute loss with fake images\n",
        "        z = torch.randn(batch_size, latent_size).cuda()\n",
        "        z = Variable(z)\n",
        "        fake_images, theta = G(z)\n",
        "        # generated labels from labeling function (here, we are showing one labeling function)\n",
        "        gen_labels_1 = lf1(fake_images, theta[:,0])\n",
        "        #gen_labels_1 = torch.nn.functional.one_hot(gen_labels_1.to(torch.int64), num_classes=nclass).float()\n",
        "        gen_labels_2 = lf2(fake_images, theta[:,1])\n",
        "        #gen_labels_2 = torch.nn.functional.one_hot(gen_labels_2.to(torch.int64), num_classes=nclass).float()\n",
        "    \n",
        "\n",
        "        gen_labels = gen_labels_1 + gen_labels_2\n",
        "        gen_labels = torch.nn.functional.one_hot(gen_labels.to(torch.int64), num_classes=nclass).float()\n",
        "        \n",
        "        gen_labels = gen_labels.view(batch_size, 10)\n",
        "        \n",
        "        outputs = D(fake_images, gen_labels)\n",
        "        \n",
        "        # We train G to maximize log(D(G(z)) instead of minimizing log(1-D(G(z)))\n",
        "        # For the reason, see the last paragraph of section 3. https://arxiv.org/pdf/1406.2661.pdf\n",
        "        g_loss = criterion(outputs, real_labels)\n",
        "        \n",
        "        # Backprop and optimize\n",
        "        # if G is trained so well, then don't update\n",
        "        reset_grad()\n",
        "        g_loss.backward()\n",
        "        g_optimizer.step()\n",
        "\n",
        "\n",
        "         #=================================================================== #\n",
        "        #                          Update Statistics                          #\n",
        "        # =================================================================== #\n",
        "        d_losses[epoch] = d_losses[epoch]*(i/(i+1.)) + d_loss.data*(1./(i+1.))\n",
        "        g_losses[epoch] = g_losses[epoch]*(i/(i+1.)) + g_loss.data*(1./(i+1.))\n",
        "        real_scores[epoch] = real_scores[epoch]*(i/(i+1.)) + real_score.mean().data*(1./(i+1.))\n",
        "        fake_scores[epoch] = fake_scores[epoch]*(i/(i+1.)) + fake_score.mean().data*(1./(i+1.))\n",
        "\n",
        "\n",
        "        if (i+1) % 200 == 0:\n",
        "            print('Epoch [{}/{}], Step [{}/{}], d_loss: {:.4f}, g_loss: {:.4f}, D(x): {:.2f}, D(G(z)): {:.2f}' \n",
        "                  .format(epoch, num_epochs, i+1, total_step, d_loss.data, g_loss.data, \n",
        "                          real_score.mean().data, fake_score.mean().data))\n",
        "            \n",
        "    # Save real images\n",
        "    if (epoch+1) == 1:\n",
        "        images = images.view(images.size(0), 1, 28, 28)\n",
        "        save_image(denorm(images.data), os.path.join(sample_dir, 'real_images.png'))\n",
        "    \n",
        "    # Save sampled images\n",
        "    fake_images = fake_images.view(fake_images.size(0), 1, 28, 28)\n",
        "    print(gen_labels)\n",
        "    save_image(denorm(fake_images.data), os.path.join(sample_dir, 'fake_images-{}.png'.format(epoch+1)))\n",
        "    \n",
        "    # Save and plot Statistics\n",
        "    np.save(os.path.join(save_dir, 'd_losses.npy'), d_losses)\n",
        "    np.save(os.path.join(save_dir, 'g_losses.npy'), g_losses)\n",
        "    np.save(os.path.join(save_dir, 'fake_scores.npy'), fake_scores)\n",
        "    np.save(os.path.join(save_dir, 'real_scores.npy'), real_scores)\n",
        "    \n",
        "    plt.figure()\n",
        "    pylab.xlim(0, num_epochs + 1)\n",
        "    plt.plot(range(1, num_epochs + 1), d_losses, label='d loss')\n",
        "    plt.plot(range(1, num_epochs + 1), g_losses, label='g loss')    \n",
        "    plt.legend()\n",
        "    plt.savefig(os.path.join(save_dir, 'loss.pdf'))\n",
        "    plt.close()\n",
        "\n",
        "    plt.figure()\n",
        "    pylab.xlim(0, num_epochs + 1)\n",
        "    pylab.ylim(0, 1)\n",
        "    plt.plot(range(1, num_epochs + 1), fake_scores, label='fake score')\n",
        "    plt.plot(range(1, num_epochs + 1), real_scores, label='real score')    \n",
        "    plt.legend()\n",
        "    plt.savefig(os.path.join(save_dir, 'accuracy.pdf'))\n",
        "    plt.close()\n",
        "\n",
        "    # Save model at checkpoints\n",
        "    if (epoch+1) % 50 == 0:\n",
        "        torch.save(G.state_dict(), os.path.join(save_dir, 'G--{}.ckpt'.format(epoch+1)))\n",
        "        torch.save(D.state_dict(), os.path.join(save_dir, 'D--{}.ckpt'.format(epoch+1)))\n",
        "\n",
        "# Save the model checkpoints \n",
        "torch.save(G.state_dict(), 'G.ckpt')\n",
        "torch.save(D.state_dict(), 'D.ckpt')"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n",
            "predicting on cuda:0..\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-15-252895bba643>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     33\u001b[0m         \u001b[0mSecond\u001b[0m \u001b[0mterm\u001b[0m \u001b[0mof\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mloss\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0malways\u001b[0m \u001b[0mzero\u001b[0m \u001b[0msince\u001b[0m \u001b[0mreal_labels\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     34\u001b[0m         '''\n\u001b[0;32m---> 35\u001b[0;31m         \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mD\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mimages\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moriginal_labels\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     36\u001b[0m         \u001b[0md_loss_real\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcriterion\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreal_labels\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     37\u001b[0m         \u001b[0mreal_score\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutputs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    887\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    888\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 889\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    890\u001b[0m         for hook in itertools.chain(\n\u001b[1;32m    891\u001b[0m                 \u001b[0m_global_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-10-9218bdfc3d80>\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, image, label)\u001b[0m\n\u001b[1;32m     54\u001b[0m       \u001b[0mcommon\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mimage\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabel\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     55\u001b[0m       \u001b[0;31m# result is the batch of probability score\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 56\u001b[0;31m       \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mD_common\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommon\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     57\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    887\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    888\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 889\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    890\u001b[0m         for hook in itertools.chain(\n\u001b[1;32m    891\u001b[0m                 \u001b[0m_global_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/container.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    117\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    118\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 119\u001b[0;31m             \u001b[0minput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    120\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    121\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    887\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    888\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 889\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    890\u001b[0m         for hook in itertools.chain(\n\u001b[1;32m    891\u001b[0m                 \u001b[0m_global_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/linear.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m     92\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     93\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 94\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mF\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     95\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     96\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mextra_repr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py\u001b[0m in \u001b[0;36mlinear\u001b[0;34m(input, weight, bias)\u001b[0m\n\u001b[1;32m   1751\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhas_torch_function_variadic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1752\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mhandle_torch_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlinear\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1753\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_C\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_nn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1754\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1755\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BR4o5XgZBHBl"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}