Skip to content
Supplemental materials for Interpretable Discovery in Large Image Datasets at WHI2018
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
data/sun-class
src
.gitignore
README.md

README.md

Interpretable Discovery in Large Image Datasets

Kiri Wagstaff, Jake Lee

in WHI 2018

This repository contains supplemental scripts and data used in the experiments presented in the paper.

Step-by-step instructions for running experiments

  1. Download the imageset - It is recommended that the image filename include the class information. The images can be in class-related subfolders; if not, it can be placed in a single subfolder (ex. /data/1/*.jpg)

  2. Preprocess the imageset - We recommend scaling/center-cropping your images to 227x227 first. We used imagemagick: $ mogrify -path imageset/# -format jpg -resize "227x227^" -gravity center -crop 227x227+0+0 +repage imageset/#/*.jpg

  3. Download/install DEMUD - Available from https://github.com/wkiri/DEMUD

  4. Extract features - Extract features from the images by using DEMUD/scripts/cnn_feat_extraction/feat_csv.py. The extracted features will be saved as a CSV, with the first column being the image name.

Note: You will need to install Caffe and to specify the trained Caffe model from which the features will be extracted. We used Caffe's pre-trained network called bvlc_reference_caffenet with a modified deploy.prototxt. The modified prototxt is available in this repository.

  1. Run DEMUD on features - Run DEMUD by adding the path to the feature CSV in demud.config at floatdatafile and running python demud.py -v --init-item=svd --k=50 It will output recon-cnn-k=50..., the reconstructed features, resid-cnn-k50..., the residual features, select-cnn-k=50..., the features of the selected image, and selections-k50.csv, a list of selected image names (in corresponding order with the other outputs).

  2. Download D&B CVPR 2016 - Available from https://lmb.informatik.uni-freiburg.de/resources/software.php

  3. Visualize the output features - Using src/demud_inversion_cvpr2016.py provided in this repo (see below for instructions)

  4. Plot discovery rates (optional) - Using src/plot/plot_exp.py provided in this repo

  5. Organize your visualizations - Using src/gen_html.py provided in this repo

Scripts

src/gen_html.py collects the results from DEMUD and Dosovitskiy and Brox inversions into a single HTML file for convenient viewing. It requires the following paths to be edited into the appropriate variables:

  • outputname - The path of the html file to be generated
  • imageset_dir - The directory containing the full imageset, but not in subdirectories
  • dosov_dir - The directory containing the inversions generated by Dosovitskiy and Brox. It assumes the following subdirectories:
    • dosov_dir/select/ - Inversions of the features in the select-cnn-*.csv output from DEMUD -v
    • dosov_dir/recon/ - Inversions of the features in the recon-cnn-*.csv output from DEMUD -v
    • dosov_dir/resid/ - Inversions of the features in the resid-cnn-*.csv output from DEMUD -v
  • demud_path - The selections-kn.csv output file from DEMUD -v
  • perfplot_path - The path to the discovery performance plot, if it was generated.

src/plot/plot_exp.py and src/plot/util_plot.py were used to generate the plots shown in the paper. plot_exp.py requires the directory containing DEMUD result folders. Note that, since these scripts were used specifically for our set of experiments, they may not be easy to use for other experiments.


src/demud_inversion_cvpr2016.py is a modified version of demo.py from Dosovitskiy and Brox, CVPR 2016. It reads each of the output CSVs from DEMUD -v/--cnn (selected features, reconstructed features, and residual features), and inverts them for visualization. The required pretrained model definitions are available here


src/build_imageset/ext_from_train.py builds the balanced ImageNet dataset used in the experiments from the ILSVRC 2012 training set. The classes to be pulled are specified in build_imageset/random_classes.txt. The files are renamed to match the pattern classID_imageNO.jpg and placed in their respective subfolders; for example, an image may be /output/n02009912/n02009912_0.jpg.


src/deploy.prototxt is a modified prototxt for the model bvlc_reference_caffenet. It is modified to perform ReLU separately instead of in-place for the fully connected layers, allowing us to extract features before ReLU is applied. It requires more memory to run, however.

Data

data/sun-class/ includes an additional "sun" class to the Mars image dataset available on Zenodo. These 21 additional images were included for the experiments presented in the paper.

You can’t perform that action at this time.