Requester tool for task design
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
scripts
src
util
.babelrc
.eslintrc.json
.gitignore
.tern-project
README.md
app.yaml
appengine_config.py
deploy.sh
fetch_data.py
index.yaml
main.py
model.py
package-lock.json
package.json
requirements.txt
serve-public.sh
serve.sh
update_eslint.sh

README.md

Sprout

For more details about the interface configurations and their evalutions, see the related paper:

Jonathan Bragg, Mausam, and Daniel S. Weld. Sprout: Crowd-Powered Task Design for Crowdsourcing. In Proceedings of the 31st ACM User Interface Software and Technology Symposium (UIST '18). 2018. To appear.

Installation

To install the front-end dependencies:

npm install

To install the back-end dependencies (for experiment logging):

  1. Install the App Engine Python SDK.
  2. Install dependencies in the project's lib directory. Note: App Engine can only import libraries from inside your project directory.
pip install -r requirements.txt -t lib

In order to access the remote API to pull logs from the backend server, obtain a key from App Engine:

  1. Go to the IAM & Admin -> Service accounts panel of the cloud console.
  2. For the App Engine default service account, create a local {FILENAME}.json key by selecting Options -> Create key, where {FILENAME} is assigned by Google.
  3. Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to this file, for instance by adding export GOOGLE_APPLICATION_CREDENTIALS="{PATH/TO/FILENAME.json}" to your ~/.bashrc.

To install utility dependencies (for preparing data files and performing analysis locally):

conda env install -f util/environment.yml -n {ENVIRONMENT_NAME}

You can also choose to use util/environment_dev.yml for useful development dependencies.

Usage

To build a development bundle:

npm run build

To build a production bundle:

npm run build-prod

To continuously watch for and re-build on changes (useful for development):

npm run watch

To run this project locally from the command line (after building the front-end):

dev_appserver.py .

Visit the application at http://localhost:8080/. See the documentation for other options for dev_appserver.py.

Getting started

Four sample starting configurations are included (and defined in src/config.js). Two of these are tutorials:

Config

Specify src/config.js to define your own tasks. For example:

export default {
  tasks: [
    [taskId]: {
      experimentPath: {path/to/experiment/file},
      answersPath: {path/to/answers/file},
      itemRootPath: {path/to/root/path/for/images},
      initialInstructions: 'instructions go here',
      tutorial: false,
    },
  ],
  experiments: {
    [experimentId]: {
      tutorial: {taskIdTutorial},
      tasks: [{taskId1}, {taskId2}, {taskId3}],
    }
  },
};

If taskIndex or tutorial are specified but not experimentId, experimentId defaults to "default".

Website usage

url parameters: /:taskIndex?/:participantIndex? all parameters:

  • systemVersion={0,1,2}
  • taskIndex={0,1,2}
  • tutorial?
  • taskId={:str}
  • experimentId={:str}
  • participantId={:str}: useful for tracking a participant outside a study

Deploying the back-end

To deploy the application:

  1. Use the Admin Console to create a project/app id. (App id and project id are identical)
  2. Deploy the application.

Fetching experiment data for analysis

  1. Export the PYTHONPATH environment variable for your Python directory, for example:
export PYTHONPATH=/usr/somedir/v3/bin/python2.7

Replace that path with the actual values for your python location. 2. Add your App Engine SDK for Python location to PYTHONPATH:

export GAE_SDK_ROOT="/usr/local/home/mydir/google_appengine"
export PYTHONPATH=${GAE_SDK_ROOT}:${PYTHONPATH}

Replace the SDK path shown above with your actual path to the App Engine SDK. 3. Export the GOOGLE_APPLICATION_CREDENTIALS environment variable as described here. 4. Run this command

python fetch_data.py PROJECT_ID

Replace the project ID above with your actual project id. Data will be stored by default in experiment_data/.

Steps 1 through 3 are from here.

Use util/analyze.py to generate analysis for experiment records.

Front-end input files

NOTE: File locations are temporary.

Experiment data

  • NOTE: To add cluster and vector fields to an experiment data file, use util/cluster.py.
  • format (in Orderly format, which can be compiled into JSONSchema):
object {
  array [
    object {
      integer id?;  # defaults to position in array
      integer cluster?;  # cluster item belongs to, based on vector embedding of worker answers
      boolean exemplar?;  # exemplar for cluster
      array [
        integer*;
      ] vector?;  # aggregate embedding for answers associated with the item
      object {
        string: path;
        object {
        } query?;
      } data;
    }*;
  ] data;  # these are the items
};

Worker answers

  • format:
array [
  object {
    object {
      string answer;
      string unclear_type;
      string unclear_reason;
      integer questionid;
    } data;
  }*;
];

Saved state

  • format:
object {
  string: instructions?  # Instructions text
  array [
    object {
      integer id;
      array [
        integer*;
      ] itemIds?;  # Ordered list of items in group
      string label;
      string name;
    }*;
  ] groups?;
  array [
    object {
      integer id;
      integer label?;
      integer group?;
      object {
        string label;
        string text;
      } reason?;
      boolean test?;
    }*;
  ] items?;
};