Discriminative Multimodal Learning via Conditional Priors in Generative Models

Code for the framework in Discriminative Multimodal Learning via Conditional Priors in Generative Models (paper).

If you use this code in your research, please cite:

@article{mancisidor2021deep,
	title={Discriminative Multimodal Learning via Conditional Priors in Generative Models},
	author={Mancisidor, Rogelio A and Kampffmeyer, Michael and Aas, Kjersti and Jenssen, Robert},
	journal={arXiv preprint arXiv:2110.04616},
	year={2021},
}

Requirements

The code for CMMD is developed in Theano. We suggest to run the code using Docker. Run the command docker pull rogelioandrade/theano_setup:v4 to pull an image with all dependencies into your local machine or to run the code in a cloud cluster.

Run mkdir output data to get a structure for the project like this:

cmmd 
   │───data
   │───output
   │───python

Otherwise you will get error messages when loading the data, saving figures etc.

Note: you can run bash build_image to build the above image and mount all folders inside cmmd.

Downloads

MNIST 2-modalities

You can download the data set from here data, or you can get the matlab code to generate the two different modalities from here code. The file is called createMNIST.m.

XRMB

You can obtain the two files for the XRMB data set here data1 and data2 or you can visit this website link to obtain the XRMB data.

Make sure to save the data files inside data

Flickr

You can obtaniened the data set here data.

MNIST-SHVN

See this link for details about how to create the data set.

Pretrained models

Reproduce the results in table 2 and 6.

MNIST: Dowload the pretranied model from here pretrained_mnist. Click the 'download' botton, then unzip the folder and save it in the output folder. Then run test_cmmd.py and choose mnist and idx=0 as arguments for test_cmmd.

XRMB: Pretrained weights for XRMB can be obtained here pretrained_xrmb. Click the 'download' botton, then unzip the folder and save it in the output folder. In this case, choose xrbm and idx=0 as arguments for test_cmmd.

Note: Weights trained on a GPU(CPU) can only be loaded again on a GPU(CPU). The pretrained weights in the above links were trained on a GPU.

Usage

Training

Make sure the requirements are satisfied in your environment, and relevant datasets are downloaded. cd into python, and run

THEANO_FLAGS=device=cuda0,floatX=float32 python -u ./train_cmmd.py --omega 0.4 --hdim_enc 2500 2500 2500 --hdim_dec 1024 1024 1024 --hdim_prior 1024 1024 1024 --zdim 50 --hdim_cls 70 70 --epochs 1001 --R 1 --outfile mnist --dset mnist

To run the code on cpu replace device=cuda0 for device=cpu.

You can play with the hyperparameters using arguments, e.g.:

• --omega: omega parameter controling the mutual information optimization.
• --dropout_rate dropout probability
• --zdim: dimension of latent variable

For all arguments see all add_argument() functions in train_cmmd.py