Skip to content
Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Annotation Game

In this game, participants are required to annotate data according to some target concept in T rounds.

Each round consists of two phases:
The participant will be required to annotate K instances from the dataset.
M annotation examples from experts will be show to the participant.

Every participant will be paid $0.01 for the first round, and get performance-contingent bonus $0.02 each round after.

Please do as well as you can!

We will qualify you for bouns in the test phase, and suggest you some expert labels for help in the teaching phase.
The bouns is given accordingly to your relative performance (a.k.a improvement) rather than your absolute performance (a.k.a. accuracy).
Therefore, the best way to maximize your gains is to do the best in each round! You would earn as much as experts do!

Task: "ピンク"

Please annotate all the "ピンク" instances with "1" and others with "0".

Features $x = (f_1, f_2, f_3)$:

  • Shape $[f_1]$: 1 = triangle, 0 = circle.
  • Border $[f_2]$: 1 = real border, 0 = dotted border.
  • Color $[f_3]$: 1 = pink, 0 = bule.

Hypothesis Space $\mathcal H = {h_1, h_1', h_2, h_2', h_3, h_3'}$

  • $h_1(x) = f_1$ (ピンク is triangle) and $h_1'(x) = 1 - f_1$ (ピンク is circle)
  • $h_2(x) = f_2$ (ピンク is real) and $h_1'(x) = 1 - f_2$ (ピンク is dotted)
  • $h_3(x) = f_3$ (ピンク is pink) and $h_1'(x) = 1 - f_3$ (ピンク is bule)


Incentivize and optimally teaching the crowd to label the data



No releases published


No packages published