Code to build VAE models that are jointly conditioned.
Switch branches/tags
Nothing to show
Clone or download
Latest commit 984f456 Dec 17, 2017
Permalink
Failed to load latest commit information.
classification Initial commit. Work by Rama Vedantam, Ian Fischer, Hernan Moraldo, a… Dec 6, 2017
datasets Fixes a dependency issue in mnista data provider. Dec 8, 2017
experiments Adds missing results_utils.py file. Dec 12, 2017
install Local fixes to imagination run paths. Dec 8, 2017
joint_vae Initial commit. Work by Rama Vedantam, Ian Fischer, Hernan Moraldo, a… Dec 6, 2017
query_files/mnist_with_attributes_dataset_iid Adds missing query files (needed them in a different naming conventio… Dec 11, 2017
scripts Merge branch 'master' of github.com:google/joint_vae Dec 8, 2017
third_party Adds missing init.py in third_party/interpolate Dec 8, 2017
utils Cleans up organization of datasets, puts all the dataset creation/dow… Dec 6, 2017
.gitignore Initial commit. Work by Rama Vedantam, Ian Fischer, Hernan Moraldo, a… Dec 6, 2017
CONTRIBUTING.md Initial commit. Work by Rama Vedantam, Ian Fischer, Hernan Moraldo, a… Dec 6, 2017
INSTRUCTIONS.md Initial commit. Work by Rama Vedantam, Ian Fischer, Hernan Moraldo, a… Dec 6, 2017
LICENSE Initial commit. Work by Rama Vedantam, Ian Fischer, Hernan Moraldo, a… Dec 6, 2017
MANIFEST.in Initial commit. Work by Rama Vedantam, Ian Fischer, Hernan Moraldo, a… Dec 6, 2017
README.md Update to the readme. Dec 17, 2017
preamble.sh Updates the README, provides facility to automatically download MNIST… Dec 7, 2017
setup.py Initial commit. Work by Rama Vedantam, Ian Fischer, Hernan Moraldo, a… Dec 6, 2017

README.md

Joint VAE Model API

Variational autoencoder for images and text


Describes the code used to create results for the following paper:

Vedantam, Ramakrishna, Ian Fischer, Jonathan Huang, and Kevin Murphy. 2017. Generative Models of Visually Grounded Imagination. arXiv [cs.LG]. arXiv. http://arxiv.org/abs/1705.10762.

Usage:

NOTE: All scripts should be run from the root directory of the project.

Basic Setup

Install basic dependencies.

cd to the root directory and run the following command.

source install/install_deps.sh

This sets up the python virtual environment for the project, and downloads necessary files for MNIST-A experiments.

Additional Data Setup

To create your own MNISTA dataset see scripts/create_affine_mnist.sh. Set appropriate paths in datasets/mnist_attributes/affine_mnist_dataset_iid.py and datasets/mnist_attributes/affine_mnist_dataset_comp.py respectively.

To download and process CELEBA dataset run scripts/process_and_write_celeba.sh. And set appropriate paths in datasets/celeba/celeba_dataset.py.

Experiments

See scripts/ for example uses of different models/ experiments reported in the paper.

  1. iclr_comp_mnista_fresh.sh: Experiments on compositional split of MNISTA. 2) iclr_mnista_fresh.sh: Experiments on iid split of MNISTA. 3) celeba_training.sh: Experiments on CelebA dataset.

Running above scripts uses slurm to run/launch jobs on clusters, so only run the above if you have access to a cluster with slurm installed on it. If you dont have access to slurm simply run the corresponding python commands in the above scripts with the appropriate parameters.


Each experiment is associated with three jobs: train, eval and imeval (imagination evaluation).

Example commands to run experiments with slurm:

  • source scripts/iclr_mnista_fresh.sh '' (to run all 3 jobs for every experiment)
  • source scripts/iclr_mnista_fresh.sh train (to launch only training jobs)
  • source scripts/iclr_mnista_fresh.sh eval (to launch only eval jobs)
    • source scripts/iclr_mnista_fresh.sh imeval (to launch only imagination eval jobs)

If you dont have access to slurm, you can see what command line arguments are used for running experiments (in the above scripts) and run those commands in bash.

Quantiative Results

See the ipython notebook experiments/iclr_results_aggregate.ipynb on how to view the imagination results after running imeval (imagination evaluation) jobs, post training.

Contributors

  • Ramakrishna Vedantam
  • Hernan Moraldo
  • Ian Fischer

Disclaimer:

This project is not an official Google project. It is not supported by Google and Google specifically disclaims all warranties as to its quality, merchantability, or fitness for a particular purpose.

Contributing:

See how to contribute.

License:

Apache 2.0.