AVB.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## COMBINING GANs AND VAEs: ADVERSARIAL VARIATIONAL BAYES (AVB)\n",
    "\n",
    "#### Giuseppe Onesto"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### VAEs and GANs description and disadvantages\n",
    "\n",
    "Variational Autoencoders (<b>VAEs</b>) and Generative Adversarial Networks(<b>GANs</b>) are two greatest methods to estimate generative models. \n",
    "More in depth, VAEs represent expressive latent variable models that can be used to learn complex probability distributions from training data; while GANs represent an adversarial process to solve the task, by simultaneously training two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. [1]\n",
    "Even though they're used reaching great results in literature, they both have some disadvantages, how explained by the authors of Adversarial Variational Bayes(AVBs)[2] models: for what concerns VAEs, the main disadvantage regards the fact that the quality of the resulting model crucially relies on the expressiveness of the inference model. Abouth GANs, it's very hard for the model to learn to generate discrete data; and their training to be stable require finding a \"Nash equilibrium of a game\", that is not as immediate. <br>\n",
    "<b>Adversarial Variational  Bayes  (AVB)</b>, is a  technique for training Variational Autoencoders with arbitrarily expressive  inference models. This is achieved by introducing an auxiliary discriminative network that allows to rephrase the maximum-likelihood problem as a two-player game, hence establishing a principled connection between VAEs and GANs.\n",
    "\n",
    "#### The AVB Model\n",
    "\n",
    "The model is an extension of VAEs model; in general VAEs are specified by a parametric generative model $p_\\theta(x|z)$\n",
    "of the visible variables given the latent variables, a prior p(z) over the latent variables and an approximate inference\n",
    "model $ q_\\phi(z|x) $ over the latent variables given the visible variables. It can be shown that: <br>\n",
    "<p style=\"text-align: center;\"> $ \\log p_\\theta(x) \\geq -KL(q_\\phi(z|x),p(z))+ E_{q\\phi(z|x)} \\log p_\\theta(x|z) $</p>\n",
    "\n",
    "When performing maximum-likelihood training, the goal is to optimize the marginal log-likelihood: <br>\n",
    "<p style=\"text-align: center;\">$ E_{p D(x)} \\log p_\\theta(x) $ , where p(D) is the  data  distribution.</p>\n",
    "\n",
    "Apart from mathematical formulas, the key point is that VAEs models quality strictly depends on the expressiveness of the inference model $ q_\\phi(z|x) $ .\n",
    "\n",
    "In the AVB model, this is handled by adding noise as additional input to the inference model, instead of adding it at the very end (as happens in VAEs), allowing the inference network to learn complex probability distributions. <br>\n",
    "We can think of AVB as a black-box inference model $ q_\\phi(z|x) $ that uses adversarial training to obtain an approximate maximum likelihood assignment $ \\theta ^* $ to $ \\theta $ and a close approximation $ q_{\\phi ^*}(z|x) $ to the true posterior $ p_{\\theta ^*}(z|x) $ \n",
    "This is shown in the below image, taken by the authors paper [2].\n",
    "\n",
    "![alt text](AVB.png)\n",
    "<p style=\"text-align: center;\"><b>Figure 1:</b> Comparison of traditional VAEs models and AVB model</p>\n",
    "\n",
    "\n",
    "To understand the link between AVB and GANs, that's the main idea of AVB starting from VAEs, we've to show the objective of the inference model presented above, that is:\n",
    " <p style=\"text-align: center;\"> $ \\max _\\theta { \\max _\\phi  {E_ {p D(x)} E_ {q_\\phi (z|x)} (\\log p(z) - \\log q_\\phi(z|x) + \\log p_\\theta(x|z))} } $ </p> \n",
    " \n",
    "The idea is to implicitly represent the term $ \\log p(z) - \\log q_\\phi(z|x) $ as the optimal value of an additional real-valued discriminative network T(x,z) that is introduced to the problem.<br>\n",
    "For more on the structure of the network and mathematical proofs of the model, please see [2].\n",
    "\n",
    "\n",
    "#### Experimental results\n",
    "\n",
    "Here I would like to show a result of the application of AVB, with respect to a VAE model (generated by a diagonal Gaussian posterior distribution), to learn a generative model. \n",
    "The AVB neural networks was trained on a very simple synthetic dataset, containing 4 data points (space of 2*2 from the image posted below):\n",
    "![alt text](synthetic.png)\n",
    "<p style=\"text-align: center;\"><b>Figure 2:</b> Images used for AVB training</p>\n",
    "\n",
    "The encoder network of AVB takes as input a data point x and a vector of Gaussian random noise and produces a latent code z. The decoder network takes as input a latent code z and produces  the  parameters  for  four  independent  Bernoulli-distributions, one for each pixel of the output image. The encoder and the decoder of VAE are parametrized as the AVB ones, but of course the encoder didn't take any noise as input.\n",
    "In the images below are shown the results obtained by the authors:\n",
    "\n",
    "![alt text](AVBvsVAE1.jpg)\n",
    "<p style=\"text-align: center;\"><b>Figure 3:</b> Distribution of VAE and AVB latent code results</p>\n",
    "\n",
    "![alt text](AVBvsVAE2.png)\n",
    "<p style=\"text-align: center;\"><b>Figure 4:</b> Comparison of VAE and AVB results</p>\n",
    "\n",
    "\n",
    "#### The AVB github project\n",
    "\n",
    "Finally, I'ld like to share the authors [GitHub project](https://github.com/LMescheder/AdversarialVariationalBayes), in which they've realized their AVB model in python, sharing the datasets used for experiments, and giving the possibility to use AVB for both variational inference and generative models. It's very interesting to play with.\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### REFERENCES\n",
    "\n",
    "[1] Goodfellow, Ian, et al. \"Generative adversarial nets.\" Advances in neural information processing systems. 2014. <br>\n",
    "[2] Mescheder, Lars, Sebastian Nowozin, and Andreas Geiger. \"Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks.\" arXiv preprint arXiv:1701.04722 (2017)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}