Caffe with heatmap regression & spatial fusion layers. Useful for any CNN image position regression task.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
cmake Merge pull request #2990 from mattdawkins/add-openblas-path Sep 2, 2015
data [example] image classification web demo Jul 12, 2014
docs information about new implemented solvers Aug 14, 2015
examples Merge pull request #3010 from darrengarvey/fixup-memory-leak-siamese Sep 2, 2015
include/caffe Caffe-heatmap Sep 28, 2015
matlab Testing code for the FLIC pretrained model Nov 8, 2015
models Add note about multfact to prototxt Mar 1, 2016
python Merge pull request #3024 from danielgordon10/python-solver-fix Sep 4, 2015
sample_source Update documentation Oct 21, 2015
scripts Fix download model binary script to get correct lines on parsing table Aug 6, 2015
src Re-center default Gaussian to 0 instead of 1 for faster network conve… Dec 1, 2015
tools Update extract_features.cpp Sep 4, 2015
.Doxyfile update doxygen config to stop warnings Sep 3, 2014
.gitignore Caffe-heatmap Sep 28, 2015
.travis.yml Travis scripts for python3 and pytest for cmake. Also fixes CUDA CMak… Jul 21, 2015
CMakeLists.txt Travis scripts for python3 and pytest for cmake. Also fixes CUDA CMak… Jul 21, 2015
CONTRIBUTING.md [docs] add CONTRIBUTING.md which will appear on GitHub new Issue/PR p… Jul 30, 2015
CONTRIBUTORS.md clarify the license and copyright terms of the project Aug 7, 2014
INSTALL.md replace bundled install instructions with link to site Feb 10, 2014
LICENSE copyright 2015 Jun 23, 2015
Makefile New make target to only build the library. Aug 14, 2015
Makefile.config.example Add commented out helpers for homebrew users Apr 2, 2015
README.md Add links to ChaLearn model and precropped images for FLIC Jan 5, 2016
caffe.cloc [fix] stop cloc complaint about cu type Sep 4, 2014
train_heatmap.sh Caffe-heatmap Sep 28, 2015

README.md

Caffe-heatmap

This is a fork of Caffe that enables training of heatmap regressor ConvNets for the general problem of regressing (x,y) positions in images.

Pretrained models

Pre-cropped images and training labels for FLIC

  • Training
  • Testing
  • Note these files require multfact=282 in both training and testing data layers

Testing instructions

matlab/pose/demo.m provides example code for running the FLIC model on a video

Training instructions

To start training:

  • Prepare your input images

  • Create two label files, one for training and another for testing, in the format:

    train/FILE.jpg 123,144,165,123,66,22 372.296,720,1,480,0.53333 0

    This is a space-delimited file where

    • the first arg is the path to your image
    • the second arg is a comma-delimited list of (x,y) coordinates you wish to regress (the coordinates in the train/FILE.jpg image space)
    • the third arg is a comma-delimited list of crops & scaling factors of the input image (in order x_left,x_right,y_left,y_right,scaling_fact). Note: These crop & scaling factors are only used to crop the mean image. You can set these to 0 if you aren't using a mean image (for mean subtraction).
    • the fourth arg is a coordinate 'cluster' (from which you have the option to evenly sample images in training). You can set this to 0.

    Example pre-cropped images and label files for FLIC are provided above.

  • Modify file paths in models/heatmap-flic-fusion/train_val.txt

  • Start training: sh train_heatmap.sh heatmap-flic-fusion 1

Supported augmentations

  • Random crop, resize, mirror and rotation

Heatmap params

  • visualise: show visualisations for crops, rotations etc (recommended for testing)
  • source: label file
  • root_img_dir: directory with images (recommend you store images on ramdisk)
  • meanfile: proto file containing the mean image(s) to be subtracted (optional)
  • cropsize: size of random crop (randomly cropped from the original image)
  • outsize: size that crops are resized to
  • multfact: label coordinates in the ground truth text file are multiplied by this (default 1)
  • sample_per_cluster: sample evenly across clusters
  • random_crop: do random crop (if false, do center crop)
  • label_height/width: width of regressed heatmap (must match net config)
  • segmentation: segment images on the fly (assumes images are in a segs/ directory)
  • angle_max: max rotation angle for training augmentation

Pose estimation-specific parameters

  • flip_joint_labels: when horizontally flipping images for augmentation, if this is set to true the code also swaps left<->right labels (this is important e.g. for observer-centric pose estimation). This assumes that the left,right joint labelsare listed consecutively (e.g. wrist_left,wrist_right,elbow_left,elbow_right)
  • dont_flip_first: This option allows you to turn off label mirroring for the first label. E.g. for labels head,wrist_right,wrist_left,elbow_right,elbow_left,shoulder_right,shoulder_left, the first joint is head and should not be swapped with wrist_right.

Notes

  • Ensure that the cropsize is set so that the crop normally covers most of the positions in the image that you wish to regress. E.g. for FLIC we prepared 256x256 cropped input images (cropped as a bounding box around the provided torso point) and used these as input images.

Paper

Please cite our ICCV'15 paper in your publications if this code helps your research:

  @InProceedings{Pfister15a,
    author       = "Pfister, T. and Charles, J. and Zisserman, A.",
    title        = "Flowing ConvNets for Human Pose Estimation in Videos",
    booktitle    = "IEEE International Conference on Computer Vision",
    year         = "2015",
  }

Caffe

Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by the Berkeley Vision and Learning Center (BVLC) and community contributors.

Check out the project site for all the details like

and step-by-step examples.

Join the chat at https://gitter.im/BVLC/caffe

Please join the caffe-users group or gitter chat to ask questions and talk about methods and models. Framework development discussions and thorough bug reports are collected on Issues.

Happy brewing!

License and Citation

Caffe is released under the BSD 2-Clause license. The BVLC reference models are released for unrestricted use.

Please cite Caffe in your publications if it helps your research:

@article{jia2014caffe,
  Author = {Jia, Yangqing and Shelhamer, Evan and Donahue, Jeff and Karayev, Sergey and Long, Jonathan and Girshick, Ross and Guadarrama, Sergio and Darrell, Trevor},
  Journal = {arXiv preprint arXiv:1408.5093},
  Title = {Caffe: Convolutional Architecture for Fast Feature Embedding},
  Year = {2014}
}