Generating Visual Representations for Zero-Shot Classification

This paper addresses the task of learning an image classifier when some categories are defined by semantic descriptions only (\eg visual attributes) while the others are defined by exemplar images as well. This task is often referred to as the Zero-Shot classification task (ZSC). Most of the previous methods rely on learning a common embedding space allowing to compare visual features of unknown categories with semantic descriptions. This paper argues that these approaches are limited as i) efficient discriminative classifiers can't be used ii) classification tasks with seen and unseen categories (Generalized Zero-Shot Classification or GZSC) can't be addressed efficiently. In contrast, this paper suggests to address ZSC and GZSC by i) learning a conditional generator using seen classes ii) generate artificial training examples for the categories without exemplars. ZSC is then turned into a standard supervised learning problem. Experiments with 4 generative models and 5 datasets experimentally validate the approach, giving state-of-the-art results on both ZSC and GZSC.

Getting Started

These instructions will get you a copy of the project up and running on your local machine for research purposes.

Prerequisites

This code requires a Python 2.7 environment.

This code is based on Tensorflow +1.0. Please follow the official instructions to install Tensorflow in your environment.

Remaining dependencies are:

Python 2.7
Tensorflow >1.0
Numpy 1.13
Sklearn 0.19
scipy 0.19

Datasets

• AWA1: Animals with Attributes 1
• AWA2: Animals with Attributes 2
• CUB: Caltech-UCSD Birds-200-2011
• APY: aYahoo & aPascal
• SUN: SUN Attribute

"Zero-Shot Learning - A Comprehensive Evaluation of the Good, the Bad and the Ugly" splits.

Usage

Preprocessing

python data/preprocessing.py

Training

GMMN

• Basic usage python gmmn.py
• Options
  • data_set: Dataset to use.
  • num_epoch: Number of epoch.
  • learning_rate: Starting learning rate.
  • noise_size: Input noise size.
  • drop_out_rate_input: Drop out input on layer.
  • drop_out_rate: Drop out on hidden layer.
  • validation: Validation stage, use validation split.
  • generalized: Generalized zero shot learning.
  • nb_img: Number of generated image feature by unseen class.
  • model_size: Number and size of hidden layers.

Autoencoder

• Basic usage python autoencoder.py
• Options
  • data_set: Dataset to use.
  • num_epoch: Number of epoch.
  • learning_rate: Starting learning rate.
  • batch_size: Batch_size size.
  • noise_size: Input noise size.
  • drop_out_rate_input: Drop out input on layer.
  • drop_out_rate: Drop out on hidden layer.
  • validation: Validation stage, use validation split.
  • generalized: Generalized zero shot learning.
  • nb_img: Number of generated image feature by unseen class.
  • encoder_size: Number and size of hidden encoder layer.
  • decoder_size: Number and size of hidden decoder layer.

Adversarial Autoencoder

• Basic usage python adversarial-autoencoder.py
• Options
  • data_set: Dataset to use.
  • num_epoch: Number of epoch.
  • learning_rate: Starting learning rate.
  • batch_size: Batch_size size.
  • noise_size: Input noise size.
  • drop_out_rate_input: Drop out input on layer.
  • drop_out_rate: Drop out on hidden layer.
  • validation: Validation stage, use validation split.
  • generalized: Generalized zero shot learning.
  • nb_img: Number of generated image feature by unseen class.
  • encoder_size: Number and size of hidden encoder layer.
  • decoder_size: Number and size of hidden decoder layer.
  • discriminator_size: Number and size of hidden discriminator layer.

AC-GAN

• Basic usage python ac_gan.py
• Options
  • data_set: Dataset to use.
  • num_epoch: Number of epoch.
  • learning_rate: Starting learning rate.
  • batch_size: Batch_size size.
  • noise_size: Input noise size.
  • drop_out_rate_input: Drop out input on layer.
  • drop_out_rate: Drop out on hidden layer.
  • validation: Validation stage, use validation split.
  • generalized: Generalized zero shot learning.
  • nb_img: Number of generated image feature by unseen class.
  • gen_size: Number and size of hidden generator layer.
  • dis_size: Number and size of hidden discriminator layer.

Results

References

If you use this work for your projects, please take the time to cite our ICCVW'17 paper :

https://hal.archives-ouvertes.fr/hal-01576222v3/document Maxime Bucher and Stéphane Herbin and Frédéric Jurie, Generating Visual Representations for Zero-Shot Classification, International Conference on Computer Vision Workshops, 2017.

@inproceedings{bucher2017generating,
  title={Generating Visual Representations for Zero-Shot Classification},
  author={Bucher, Maxime and Herbin, St{\'e}phane and Jurie, Fr{\'e}d{\'e}ric}
  booktitle={International Conference on Computer Vision Workshops},
  year={2017},
  organization={IEEE}
}

License

Code are released under the GPLv3 license for non-commercial and research purposes only. For commercial purposes, please contact the authors.

Acknowledgements

This work has been conducted at ONERA and GREYC CNRS, with the support of "Région Normandie".