Introduction

This repository implements Lee et al., Hierarchical Novelty Detection for Visual Object Recognition, CVPR 2018 in PyTorch.

@inproceedings{lee2018hierarchical,
  title={Hierarchical Novelty Detection for Visual Object Recognition},
  author={Lee, Kibok and Lee, Kimin and Min, Kyle and Zhang, Yuting and Shin, Jinwoo and Lee, Honglak},
  booktitle={CVPR},
  year={2018}
}

Dependencies

• Python 3.6.2 (or https://www.anaconda.com/download/)
• PyTorch 0.3.0.post4 (since Pytorch frequently updates their libraries, our code would not work if you have a different version.)
• torchvision 0.2.0
• NumPy 1.14.0
• SciPy 0.19.1 (to load features in mat format for AWA and CUB experiments)
• h5py 2.7.0 (to save features and random numbers)
• matplotlib 2.0.2 (to plot known-novel class accuracy curve)
• NLTK 3.2.4 (to convert WordNet synsets to offset ids for CUB experiments)

Data

You may download either raw images or ResNet-101 features. If you download ResNet-101 features, place them in datasets/{dataset}/. ({dataset} = ImageNet, AWA2, CUB)

ImageNet

• Raw images
  • Move ILSVRC 2012 train to datasets/ImageNet/train/.
    • e.g., an image should be found in datasets/ImageNet/train/n01440764/n01440764_18.JPEG.
  • Move ILSVRC 2012 val to datasets/ImageNet/known/.
    • ILSVRC 2012 validation dataset is not sorted. You can move validation images to labeled subfolders using [this script].
  • Move ImageNet Fall 2011 to datasets/ImageNet/novel/.
• ResNet-101 features [train] [val] [known] [novel]

AWA, CUB

• ResNet-101 features

WordNet

You do not have to download the files, but we provide the source of them for your reference.

• WordNet
• WordNet is-a relationship

Preparation

Taxonomy

Run sh scripts/preparation.sh {d}. ({d} = imagenet_full, imagenet, awa2, cub)

• {d} = imagenet_full if you have raw images
• {d} = imagenet if you have ResNet-101 features

Output files are in taxonomy/{dataset}/.

You can download pre-built taxonomies [here].

Feature extraction (ImageNet) or conversion (AWA, CUB)

Run sh scripts/feature.sh {d}. ({d} = imagenet, awa2, cub)

Output files are in datasets/{dataset}/.

If you have ResNet-101 features for ImageNet, skip this.

Train, test

Run sh scripts/train.sh {d} {m}. ({d} = imagenet, awa2, cub, {m} = relabel, td, loo, td+loo)

Output files are in train/.

You can download models reported in the paper [here].

Note

• The code keeps all random numbers and final models. For new experiment, you may remove the following if exist:

datasets/{dataset}/balanced_order_{:d}.h5
datasets/{dataset}/relabels_{:d}.h5
train/

• The code can produce results in hierarchical measures. To see them, build the taxonomy with additional argument, e.g., python build_taxonomy.py ImageNet detailed