Skip to content
Click Here: Human-Localized Keypoints as Guidance for Viewpoint Estimation
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
caffe-r4cnn @ 947f1a1
view_estimation_correspondences Added random keypoint class and image-feature-to-keypoint-weight expe… May 17, 2017
.gitignore Added caffe-r4cnn submodule May 12, 2017

Click Here: Human-Localized Keypoints as Guidance for Viewpoint Estimation

Ryan Szeto and Jason J. Corso, University of Michigan

Using this code

If you use this work in your research, please cite the following paper:

  author = "Szeto, Ryan and Corso, Jason J.",
  title = "Click Here: Human-Localized Keypoints as Guidance for Viewpoint Estimation",
  booktitle = "IEEE International Conference on Computer Vision (ICCV)",
  month = "Oct",
  year = "2017"

I ran this code from scratch, so it should work. However, feel free to contact me at szetor [at] umich [dot] edu if you have trouble.


This code implements the work described in the arXiv report "Click Here: Human-Localized Keypoints as Guidance for Viewpoint Estimation". It extends the Render for CNN project by generating semantic keypoint data alongside rendered (or real) image data. This code lets you generate the training and testing data that we used in our paper's experiments, as well as reproduce the numbers presented in the paper.

Please note that this code is insanely inefficient storage-wise, since it stores dense keypoint maps and keypoint class vectors on disk as LMDBs. Excluding code and model weights, you only need about 20 GB to run our pre-trained models on the PASCAL 3D+ test set, but you need about 3 TB for our entire training process over synthetic and real image examples. We are implementing a more efficient version of this code in TensorFlow, which might be made available someday...

Reproducing our results

In this project, we have included the pre-trained models used to produce the results in the paper. This section outlines how to run them.

Set up dataset

First, install the PASCAL 3D+ dataset as follows. The Bash scripts in this readme assume you are starting from this project's root directory unless otherwise noted.

cd datasets

Set up weights

The weights to our models are included in an external file available here. Download this folder to the project root, and then extract the contents to the demo_experiments folder:

tar -xzvf ch-cnn-model-weights.tar.gz -C demo_experiments

Set up Caffe

Our code uses a customized version of Caffe. It is based on Caffe RC1, and the main difference is that it includes the custom layers from the Caffe version used by Render for CNN (their Caffe source code is available here). To initialize the submodule, run the following commands:

git submodule init
git submodule update

Then use whatever means you wish (Make/CMake/fellow grad student) to install the customized Caffe. Remember the installation path when you set up the global variables.

Set up global variables

Copy the example global variables file, edit the paths as instructed, and propagate the variables to the demo experiment setups.

### MODIFY ###
python view_estimation_correspondences/eval_scripts/

Generate PASCAL 3D+ LMDBs

Generate the test instance data, then generate the test LMDBs.

cd view_estimation_correspondences
python --pascal_test_only
python --pascal_test_only

Run experiments

Run the evaluation code on our demo models (located in demo_experiments). It takes about an hour on a NVIDIA GeForce GTX 980 Ti GPU for each experiment, so I recommend running this overnight and/or on a cluster environment, if possible.

cd view_estimation_correspondences/eval_scripts
python 67 6000 --demo --cache_preds
python 68 0 --demo --cache_preds
python 70 2000 --demo --cache_preds
python 71 0 --demo --cache_preds
python 72 0 --demo --cache_preds
python 73 0 --demo --cache_preds
python 78 0 --demo --cache_preds
python 80 0 --demo --cache_preds
python 81 0 --demo --cache_preds
python 82 0 --demo --cache_preds
python 83 0 --demo --cache_preds
python 84 0 --demo --cache_preds
python 85 0 --demo --cache_preds
python 86 0 --demo --cache_preds
python 87 0 --demo --cache_preds
python 89 2000 --demo --cache_preds
python 90 2000 --demo --cache_preds
python 93 2000 --demo --cache_preds
python 94 2000 --demo --cache_preds
python 98 4400 --demo --cache_preds
python 99 2000 --demo --cache_preds
python 100 2000 --demo --cache_preds

Results are stored in demo_experiments/<exp_num>/evaluation.

Generate visualizations

The above commands cache the scores for each rotation angle, which can be compared with the script. The commands below compare the predictions of fine-tuned Render for CNN and our CH-CNN model.

# $PROJ_ROOT is the location of the root of this project.
cd view_estimation_correspondences/eval_scripts
python 6932 \
	$PROJ_ROOT/demo_experiments/000067/evaluation/cache_6000.pkl R4CNN \
	$PROJ_ROOT/demo_experiments/000070/evaluation/cache_2000.pkl CH-CNN

Results are stored under $PROJ_ROOT/view_estimation_correspondences/eval_scripts/visualizations/qualitative_comparison.

Generate error distribution plots

The distribution of errors for a model can be visualized with the script. The commands below generate this plot for fine-tuned Render for CNN and our CH-CNN model.

# $PROJ_ROOT is the location of the root of this project.
python \
	$PROJ_ROOT/demo_experiments/000067/evaluation/cache_6000.pkl R4CNN
python \
	$PROJ_ROOT/demo_experiments/000070/evaluation/cache_2000.pkl CH-CNN

Results are stored under $PROJ_ROOT/view_estimation_correspondences/eval_scripts/visualizations/error_distribution.

Generating training data

This section describes how to generate synthetic and real image training data with our code. Before you execute the steps below, make sure you have set up the PASCAL 3D+ dataset, Caffe and global variables as described in "Reproducing our results".

Set up datasets

You will need to download some auxiliary data and save the .zip files in the datasets folder. First, you need to download the following synsets from ShapeNet: 02924116 (buses), 02958343 (cars), and 03790512 (motorcycles). Then, download our ShapeNet keypoints dataset here. Finally, run the extraction scripts below:

cd datasets

Render synthetic images and keypoint information

Compile mex code

cd render_pipeline/kde/matlab_kde_package/mex
matlab -nodisplay -r "makemex; quit;"

Generate viewpoint and truncation distributions with KDE

cd render_pipeline/kde
matlab -nodisplay -r "run_sampling; quit;"

Render, crop, and overlay backgrounds

This takes many days on multiple cores. See the file for tips on how to make this as fast as possible.

cd render_pipeline

Generate training and testing LMDBs

This takes at least a day on multiple cores.

cd view_estimation_correspondences

Training our models

This section describes how to create and evaluate the models from our paper.

Download R4CNN weights

Run the script to download the R4CNN weights from the original authors.

cd caffe_models
./  # Run again for the checksum

Create a training run

The script in the view_estimation_correspondences folder provides an interface to automatically generate a training run (a.k.a. experiment). For example, to create an experiment to train our full CH-CNN model, run these commands:

cd view_estimation_correspondences
python CH-CNN

This creates a folder for the experiment under experiments; it is named with the experiment number and the timestamp for the date when it was generated. The script prompts you to add notes for the experiment, which are saved in under the newly-created experiment folder.

Other model names such as R4CNN, fixed_weight_map_uniform, and CH-CNN_kpm_only can be used in place of CH-CNN. To see all available options, read the main function in

By default, models are initialized with R4CNN weights and trained on synthetic data. The model weights and training/evaluation sets can be overridden by passing the following options to

  • --pascal: Train and evaluate on the PASCAL 3D+ data
  • --init_weight_path <path_to_caffemodel_file>: Use the given caffemodel file to initialize the model weights

Running and resuming an experiment

To start a training run from the beginning, run the script. Below is an example of running experiment 1 (000001):

cd train
python --exp_num 1

To resume an experiment from the latest solver state, run the same script with the --resume flag:

python --exp_num 1 --resume

Progress logs are stored in experiments/<exp_folder>/progress.

Monitoring training progress

From the progress plots, you can generate plots to visualize training progress. To do this, run the script:

cd train
python plot_training_progress,py <exp_num>

A plot of the training losses, validation losses, and angle-wise validation accuracies will be created in experiments/<exp_folder>/progress.

Training progress server

This project includes a web server that you can use to track your experiments. To run this module, you need the Python package Bottle, which can be obtained with the following command:

pip install bottle

To run the server, use the following commands:

cd train/progress_web_server
python &

Note that this only fetches existing files; it does not regenerate plots or evaluations automatically. To automatically regenerate plots, run the following command from the project root directory:

for file in `ls experiments`; do exp_num=`echo ${file:0:6} | sed -e 's/^0*//g'`; python "train/" $exp_num; done

Evaluating a trained model

A trained model can be evaluated with the same evaluation script as mentioned in "Run experiments", just without the --demo flag.

cd view_estimation_correspondences/eval_scripts
python <exp_num> <iter_num> --cache_preds

iter_num refers to the iteration number of the desired snapshot. --cache_preds is an optional flag that tells the script to save the angle scores to disk. This is useful for visualization (see "Generate visualizations").


I would like to thank Hao Su and Charles R. Qi for providing their Render for CNN code.

You can’t perform that action at this time.