Code for the chat interface used to collect the VisDial dataset on AMT
JavaScript CSS HTML Python
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
.gitignore Initial commit Dec 22, 2016

VisDial AMT Chat

Source for the two-person chat interface used to collect the VisDial dataset ( on Amazon Mechanical Turk. A demo is available here (open in two separate tabs to be paired for conversation).

VisDial AMT Interface

How things work

The real-time chat interface is built using Node.js and We use Redis to maintain a pool of images for live HITs and all data finally resides in a MySQL database.

A database table stores images from the COCO dataset each with a randomly picked caption. A batch of images from this table are then pushed to a Redis list for launching HITs. The web server corresponding to the chat interface pairs two AMT workers, assigns them roles (questioner or answerer) and shows corresponding interface, picks an image from the Redis list to collect data on and saves their conversation in the database, also marking that image as 'complete' once the HIT is done. This happens in parallel so workers aren't left waiting, and the server ensures workers have unique ids. Disconnects are handled gracefully — remaining worker is asked to continue asking questions or providing facts (captions) up to 10 messages. Once the HITs are complete, scripts in mturk_scripts/approve can be used to review, approve, reject HITs and pay workers.


Pops images from the Redis list, retrieves relevant data from the MySQL tables, renders the chat interface and pairs workers on AMT, saves submitted data to MySQL tables

  • Install MySQL, Redis, Node
  • Install dependencies using npm install (from the nodejs folder)
  • Copy over example.config.js to config.js, and set MySQL and Redis credentials
  • Create a symbolic link from static/dataset to /path/to/mscoco/images/
  • Update app path (from vhost) in index.html line 276
  • Running node index.js should now serve the interface at

MTurk Scripts

Scripts to set up MySQL database, populate Redis list, and approve/reject HITs

  • Copy over example.config.json to config.json, and set MySQL credentials. from_timestamp is unix timestamp before launching a batch of HITs
  • and create MySQL tables, populate it with COCO images and captions and generate the Redis list for a batch of HITs
  • launches HITs on AMT

Approving/Rejecting HITs (mturk_scripts/approve)

  • Copy over example.config.json to config.json, and set MySQL credentials
  • Copy over to, and set AMT credentials
  • gets completed HITs and saves them to amthitsQues.csv and amthitsAns.csv for review
  • Once HITs have been reviewed in the CSV files (by changing 'notApprove' to 'reviewReject' or 'approve'), run to mark approved HITs for payment and, to reject HITs, and finally to process payments



Parts of this code (MTurk scripts) are adapted from @jcjohnson's simple-amt project.