Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Human-Powered Data Analysis with Mechanical Turk
Ruby JavaScript

This branch is 14 commits behind master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
app
config
db
doc
lib
log
public
script
test
vendor
.gitignore
.travis.yml
Gemfile
Gemfile.lock
LICENSE
README.md
Rakefile
config.ru

README.md

Clockwork Raven Build Status Code Climate

Human-Powered Data Analysis

Clockwork Raven is a web application that allows users to easily submit data to Mechanical Turk for manual review, and then analyze that data. Clockwork Raven is actively used at Twitter to gather tens of thousands of judgments from Mechanical Turk users weekly.

What it is good for

Clockwork Raven is designed for individuals, or organizations that share a single Mechanical Turk account. It allows users to authenticate with LDAP or with accounts created through Clockwork Raven. Users can then upload data, design forms to send to Mechanical Turk through a simple, drag-and-drop interface, submit these forms to Mechanical Turk and review the results.

Administrators can separate users into "privileged" and "unprivileged" manually or based on LDAP groups. Unprivileged uses are not allowed to spend money, but can submit evaluations to the Mechanical Turk sandbox to test out the system, and can design evaluations and ask a privileged user to submit the evaluation on his or her behalf.

To help control the quality of responses, Clockwork Raven allows you to restrict the tasks you send to Mechanical Turk to only those users who Mechanical Turk has deemed "Categorization Masters", or to only those users who you have marked as Trusted in Clockwork Raven.

What it is not good for

Clockwork Raven is designed for situations where everyone who as access to the system is relatively trusted. Because the form designer allows users to use arbitrary HTML, anyone with access to the system could execute an XSS attack and compromise the system.

However, Clockwork Raven will not give access to users unless they are part of white-listed LDAP groups or have been explicitly grated access. Users cannot create accounts for themselves.

Setup

  1. Check out the code from https://github.com/twitter/clockworkraven
  2. Requirements:
    1. Make sure the machine that you're using has Ruby 1.9.3 installed. The easiest way to install and manage Ruby is with RVM.
    2. You'll need the RubyGem "bundler" installed, and then just run "bundle install" from the Clockwork Raven directory to install all of the libraries needed by Clockwork Raven.
    3. Clockwork Raven uses Resque to run tasks in the background. Resque requires a Redis server -- see Resque's instructions for installing Redis. By default Clockwork Raven assumes your Redis server is running on localhost:6379. If this isn't the case, edit config/resque.yml.
    4. In a production environment (e.g. any environment where Clockwork Raven will be accessible to users), it should be run over SSL to protect users' credentials when they log in. If you don't use SSL, these credentials will be sent over the network in the clear!
  3. Configure:

    1. Generate a secret key. Copy config/secret.example.yml to config/secret.yml. Then, run rake secret and copy the output to config/secret.yml.
    2. Copy config/database.example.yml to config/database.yml and modify it to point to your MySQL database. Currently, Clockwork Raven only supports MySQL.
    3. Copy config/mturk.example.yml to config/mturk.yml. Follow the instructions in that file to connect Clockwork Raven to your Mechanical Turk account.
    4. Configure authentication:

      LDAP Authentication

      LDAP authentication is the recommended way to manage account in Clockwork Raven. If your LDAP server supports SSL/TLS, copy config/auth.example_ldap_encrypted.yml to config/auth.yml. If your LDAP server does not, copy config/auth.example_ldap_unencrypted.yml to config/auth.yml. Follow the instructions in that file to connect Clockwork Raven to your LDAP server and grant access to specific LDAP groups and users.

      Password Authentication

      If you can't use an LDAP server, you can configure Clockwork Raven use "password authentication," which will allow you to manually create accounts. Copy config/auth.example_password.yml to config/auth.yml. Then, you can create accounts by running "rake users:add" and change passwords with "rake users:change_password". Note that you will need to set up your database (explained below) before using these rake tasks.

  4. Set up the database. If the databases you configured Clockwork Raven to use in config/database.yml do not exist, run "rake db:create" to create them. Then, run "rake db:structure:load" to load the database structure into your database.
  5. Start up the background workers. Just run rake raven:resque to start up 4 background workers. You can start up more background workers by passing an argument to the rake task: rake raven:resque[16] will start up 16 background workers.
  6. Run the server. To run the server, run "rails s".

Documentation

Documentation is available on the wiki.

Contact

Follow @clockworkraven for updates and notifications. Submit bug report and feature requests to the issue tracker.

Join the mailing list, twitter-clockworkraven@googlegroups.com, on Google Groups to ask questions and discuss development.

Roadmap

We would love any help adding ideas or implementing them!

  • JSON/REST API
  • Provide the option to have multiple Mechanical Turk users complete each task
  • Provide in-depth analytics about workers and automate the process of choosing trusted workers.

Contributing

To contribute to Clockwork Raven, fork the repo and, make your changes, and submit a pull request. All pull requests should be against *-wip branches. Nothing gets committed/merged directly to master. To merge your pull request, you'll need to include appropriate documentation and tests. Get in touch if you have any questions about what you need to do to get your contributions accepted.

Authors

Thanks for assistance and contributions:

License

Copyright 2012 Twitter, Inc.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

Something went wrong with that request. Please try again.