This page describes the Thrift API for Clockwork Raven and how to use it.
Thrift is an interface description language that is used for communication between services. It abstracts away the complexities of serialization/deserialization and implementing consistent data types in the languages of the services. More information including examples can be found at the Apache Thrift website.
The Thrift API for Clockwork Raven exposes a server through which clients can submit task requests to be sent to Mechanical Turk and also check statuses of existing tasks. Currently the Thrift definition for the API can be found in /lib/human_eval.thrift. It defines two main services, submitTask, which adds a task to an existing evaluation (specified by name) and submits that task to Mechanical Turk, and fetchAnnotations, which given a list of task identifiers, fetches statuses for each task from Mechanical Turk.
To start the server, run the /lib/daemons/human_eval_thrift_daemon.rb script; this will run the server as a daemon.
Setting up a client amounts to running the Thrift compiler on the interface description (/lib/human_eval.thrift) to generate the type definitions in the language of the client, and then implementing a client program to talk to the server.
A simple client in Ruby may look like:
transport = Thrift::BufferedTransport.new(Thrift::Socket.new('127.0.0.1', 9090)) protocol = Thrift::BinaryProtocol.new(transport) client = HumanEvalTaskManager::Client.new(protocol) transport.open() params = HumanEvalFetchAnnotationParams.new params.taskIdList =  res = client.fetchAnnotations(params)
HumanEvalTaskis the name field of the evaluation to add the task to. Names are currently unique, so this is fine.
HumanEvalSubmitTaskParamsis confusing. The reason it's confusing is that whether or not a task should be submitted to Mechanical Turk is determined by its evaluation; an evaluation is either production or sandboxed (see prod field in evaluation.rb). The current behavior is that a task will only be submitted to Mechanical Turk if the evaluation is a production evaluation and
TaskStatus.COMPLETEis a misleading name. There are two interpretations: 1) the task has all of its questions answered by a Turker, and 2) the submitter has approved the Turker's answers; the Thrift API chooses the latter interpretation. This means that a task can be "complete", as in it has each question answered, but it'll still be
TaskStatus.PENDINGuntil the submitter approves the answers.