A wrapper of the COCOeval class for extended keypoint error estimation analysis.
Branch: release
Clone or download
matteorr Merge pull request #16 from fran6co/patch-2
Make it compile on windows
Latest commit 1e49416 Oct 12, 2018


coco-analyze Repository

This repository contains the code release from the paper Benchmarking and Error Diagnosis in Multi-Instance Pose Estimation.

If you find this work useful please cite our paper:

author = {Ronchi, Matteo Ruggero and Perona, Pietro},
title = {Benchmarking and Error Diagnosis in Multi-Instance Pose Estimation},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
month = {Oct},
year = {2017}

Important Content:

  • pycocotools/COCOanalyze.py: wrapper of the COCOeval class for multi-instance keypoint estimation error analysis.
  • COCOanalyze_demo.ipynb: ipython notebook showing how to use COCOanalyze as a standalone class.
  • analysisAPI: API using COCOanalyze for an extended analysis.
  • run_analysis.py: script generating a pdf summary of the extended analysis.


Use the Makefile to install the coco-analyze api:

  • make all will compile and install locally. (RECOMMENDED)
  • make install will install the api to the Python site-packages folder. NOTE This might override your current pycocotools installation.


To run the extended multi-instance keypoint estimation error analysis: update the paths of the detections and annotations and execute the command line.

[annFile]  -> ./annotations/keypoints_val2014.json
[dtsFile]  -> ./detections/fakekeypoints100_keypoints_val2014_results.json
[saveDir]  -> ./results/fakekeypoints100
[teamName] -> fakekeypoints100
[version]  -> 1.0
$ python run_analysis.py [annFile] [dtsFile] [saveDir] [teamName] [version]


  • A summary file called [teamName]_performance_report.tex will be created once the analysis is complete.
  • All the generated plots are stored using [saveDir] as the base directory.
  • Additional std_output information regarding the analysis can be found in the text files named std_out.txt.

Automatically Generated Performance Reports

You can find examples of the reports generated by the analysis code:


  • The ./pycocotools/COCOeval class contained in this repository is a modified version of the original mscoco COCOeval class.
  • The duration of the full analysis depends on the number of detections and size of the ground-truth split.
  • You can comment out parts of run_analysis.py to run the analysis only for specific types of error.
  • Set USE_VISIBILITY_FOR_PLOTS=True in localizationErrors.py if during the analysis you wish to visualize only the keypoints whos visibility flag is 1 (visible but occluded), or 2 (visible). Check issue #14 for more details.