Skip to content
No description, website, or topics provided.
Python Jupyter Notebook Shell
Branch: master
Clone or download

Latest commit

Fetching latest commit…
Cannot retrieve the latest commit at this time.


Type Name Latest commit message Commit time
Failed to load latest commit information.

VizWiz Captions Evaluation

Code for the VizWiz API and evaluation of generated captions.

View the tutorial Jupyter Notebook.


  • python 3
  • java 1.8.0 (for caption evaluation)



  • demo_vizwiz_caption_evaluation.ipynb (tutorial notebook)


  • This file contains the VizWiz API class that can be used to load VizWiz dataset JSON files and analyze them.


  • train.json (VizWiz-Captions training set)
  • val.json (VizWiz-Captions validation set)
  • Dataset shares the same data format as MS COCO.


  • fake_caption_val.json (an example of fake results for running demo)
  • Dataset shares the same data format as MS COCO.

./vizwiz_eval_cap: The folder where all caption evaluation codes are stored.

  • This file includes the VizWizEvalCap class that can be used to evaluate results on VizWiz.
  • tokenizer: Python wrapper of Stanford CoreNLP PTBTokenizer
  • bleu: Bleu evalutation codes
  • rouge: Rouge-L evaluation codes
  • cider: CIDEr evaluation codes
  • spice: SPICE evaluation codes


  • The primary VizWiz API is standalone.
  • Download annotation files.
  • For caption evaluation, you will first need to download the Stanford CoreNLP 3.6.0 code and models for use by SPICE. To do this, run: ./
    • To run shell scripts in Windows, you can setup Windows Subsystem for Linux.
    • The command for Windows will then be bash
  • Note: SPICE will try to create a cache of parsed sentences in ./vizwiz_eval_cap/spice/cache/. This dramatically speeds up repeated evaluations. The cache directory can be moved by setting 'CACHE_DIR' in ./vizwiz_eval_cap/spice. In the same file, caching can be turned off by removing the '-cache' argument to 'spice_cmd'.




This work is closely adapted from MS COCO API and MS COCO Caption Evaluation API.

Original Developers

  • Xinlei Chen (CMU)
  • Hao Fang (University of Washington)
  • Tsung-Yi Lin (Cornell)
  • Ramakrishna Vedantam (Virgina Tech)

Original Acknowledgements

  • David Chiang (University of Norte Dame)
  • Michael Denkowski (CMU)
  • Alexander Rush (Harvard University)
You can’t perform that action at this time.