Interpretable Discovery in Large Image Datasets

Kiri Wagstaff, Jake Lee

in WHI 2018

This repository contains supplemental scripts and data used in the experiments presented in the paper.

Step-by-step instructions for running experiments

1. Download the imageset - It is recommended that the image filename include the class information. The images can be in class-related subfolders; if not, it can be placed in a single subfolder (ex. /data/1/*.jpg)

2. Preprocess the imageset - We recommend scaling/center-cropping your images to 227x227 first. We used imagemagick: $ mogrify -path imageset/# -format jpg -resize "227x227^" -gravity center -crop 227x227+0+0 +repage imageset/#/*.jpg

3. Download/install DEMUD - Available from https://github.com/wkiri/DEMUD

4. Extract features - Extract features from the images by using DEMUD/scripts/cnn_feat_extraction/feat_csv.py. The extracted features will be saved as a CSV, with the first column being the image name.

Note: You will need to install Caffe and to specify the trained Caffe model from which the features will be extracted. We used Caffe's pre-trained network called bvlc_reference_caffenet with a modified deploy.prototxt. The modified prototxt is available in this repository.

5. Run DEMUD on features - Run DEMUD by adding the path to the feature CSV in demud.config at floatdatafile and running python demud.py -v --init-item=svd --k=50 It will output recon-cnn-k=50..., the reconstructed features, resid-cnn-k50..., the residual features, select-cnn-k=50..., the features of the selected image, and selections-k50.csv, a list of selected image names (in corresponding order with the other outputs).

6. Download D&B CVPR 2016 - Available from https://lmb.informatik.uni-freiburg.de/resources/software.php

7. Visualize the output features - Using src/demud_inversion_cvpr2016.py provided in this repo (see below for instructions)

8. Plot discovery rates (optional) - Using src/plot/plot_exp.py provided in this repo

9. Organize your visualizations - Using src/gen_html.py provided in this repo

Scripts

src/gen_html.py collects the results from DEMUD and Dosovitskiy and Brox inversions into a single HTML file for convenient viewing. It requires the following paths to be edited into the appropriate variables:

• outputname - The path of the html file to be generated
• imageset_dir - The directory containing the full imageset, but not in subdirectories
• dosov_dir - The directory containing the inversions generated by Dosovitskiy and Brox. It assumes the following subdirectories:
  • dosov_dir/select/ - Inversions of the features in the select-cnn-*.csv output from DEMUD -v
  • dosov_dir/recon/ - Inversions of the features in the recon-cnn-*.csv output from DEMUD -v
  • dosov_dir/resid/ - Inversions of the features in the resid-cnn-*.csv output from DEMUD -v
• demud_path - The selections-kn.csv output file from DEMUD -v
• perfplot_path - The path to the discovery performance plot, if it was generated.

---

src/plot/plot_exp.py and src/plot/util_plot.py were used to generate the plots shown in the paper. plot_exp.py requires the directory containing DEMUD result folders. Note that, since these scripts were used specifically for our set of experiments, they may not be easy to use for other experiments.

---

src/demud_inversion_cvpr2016.py is a modified version of demo.py from Dosovitskiy and Brox, CVPR 2016. It reads each of the output CSVs from DEMUD -v/--cnn (selected features, reconstructed features, and residual features), and inverts them for visualization. The required pretrained model definitions are available here

---

src/build_imageset/ext_from_train.py builds the balanced ImageNet dataset used in the experiments from the ILSVRC 2012 training set. The classes to be pulled are specified in build_imageset/random_classes.txt. The files are renamed to match the pattern classID_imageNO.jpg and placed in their respective subfolders; for example, an image may be /output/n02009912/n02009912_0.jpg.

---

src/deploy.prototxt is a modified prototxt for the model bvlc_reference_caffenet. It is modified to perform ReLU separately instead of in-place for the fully connected layers, allowing us to extract features before ReLU is applied. It requires more memory to run, however.

Data

data/sun-class/ includes an additional "sun" class to the Mars image dataset available on Zenodo. These 21 additional images were included for the experiments presented in the paper.