Skip to content
This repository

Thrift API 

vlam21 edited this page · 8 revisions

Introduction

This page describes the Thrift API for Clockwork Raven and how to use it.

What Thrift is

Thrift is an interface description language that is used for communication between services. It abstracts away the complexities of serialization/deserialization and implementing consistent data types in the languages of the services. More information including examples can be found at the Apache Thrift website.

What the Thrift API does

The Thrift API for Clockwork Raven exposes a server through which clients can submit task requests to be sent to Mechanical Turk and also check statuses of existing tasks. Currently the Thrift definition for the API can be found in /lib/human_eval.thrift. It defines two main services, submitTask, which adds a task to an existing evaluation (specified by name) and submits that task to Mechanical Turk, and fetchAnnotations, which given a list of task identifiers, fetches statuses for each task from Mechanical Turk.

How to use the Thrift API

Starting the server

To start the server, run the /lib/daemons/human_eval_thrift_daemon.rb script; this will run the server as a daemon.

Setting up a client

Setting up a client amounts to running the Thrift compiler on the interface description (/lib/human_eval.thrift) to generate the type definitions in the language of the client, and then implementing a client program to talk to the server.

A simple client in Ruby may look like:

transport = Thrift::BufferedTransport.new(Thrift::Socket.new('127.0.0.1', 9090))
protocol = Thrift::BinaryProtocol.new(transport)
client = HumanEvalTaskManager::Client.new(protocol)

transport.open()

params = HumanEvalFetchAnnotationParams.new
params.taskIdList = [9001]

res = client.fetchAnnotations(params)

Notes on submitTask

  • The humanEvalTaskType field in HumanEvalTask is the name field of the evaluation to add the task to. Names are currently unique, so this is fine.
  • The doSubmitToProduction field in HumanEvalSubmitTaskParams is confusing. The reason it's confusing is that whether or not a task should be submitted to Mechanical Turk is determined by its evaluation; an evaluation is either production or sandboxed (see prod field in evaluation.rb). The current behavior is that a task will only be submitted to Mechanical Turk if the evaluation is a production evaluation and doSubmitToProduction is true.

Notes on fetchAnnotations

  • TaskStatus.COMPLETE is a misleading name. There are two interpretations: 1) the task has all of its questions answered by a Turker, and 2) the submitter has approved the Turker's answers; the Thrift API chooses the latter interpretation. This means that a task can be "complete", as in it has each question answered, but it'll still be TaskStatus.PENDING until the submitter approves the answers.
  • The keys in the task results map are question labels (i.e. "Please explain your answers" for a free-response question, "Is this spam?" for a multiple-choice question). The values for a free-response question are the free-text responses; the values for a multiple-choice question are the multiple choice option labels (i.e. "Yes" or "No").
Something went wrong with that request. Please try again.