Skip to content
Incentivize and optimally teaching the crowd to label the data
Jupyter Notebook Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.ipynb_checkpoints
imgs
results
Annotation Game.ipynb
Pedagogical Game.ipynb
Towards Resolving Unidentifiability.ipynb
plotall.sh
pvc.py
pvc.pyc
readme.md
simulation.py
simulation.pyc
testbed.py
utils.py
utils.pyc

readme.md

Annotation Game

In this game, participants are required to annotate data according to some target concept in T rounds.

Each round consists of two phases:
Test:
The participant will be required to annotate K instances from the dataset.
Teaching:
M annotation examples from experts will be show to the participant.

Every participant will be paid $0.01 for the first round, and get performance-contingent bonus $0.02 each round after.

Please do as well as you can!

We will qualify you for bouns in the test phase, and suggest you some expert labels for help in the teaching phase.
The bouns is given accordingly to your relative performance (a.k.a improvement) rather than your absolute performance (a.k.a. accuracy).
Therefore, the best way to maximize your gains is to do the best in each round! You would earn as much as experts do!


Task: "ピンク"

Please annotate all the "ピンク" instances with "1" and others with "0".

Features $x = (f_1, f_2, f_3)$:

  • Shape $[f_1]$: 1 = triangle, 0 = circle.
  • Border $[f_2]$: 1 = real border, 0 = dotted border.
  • Color $[f_3]$: 1 = pink, 0 = bule.

Hypothesis Space $\mathcal H = {h_1, h_1', h_2, h_2', h_3, h_3'}$

  • $h_1(x) = f_1$ (ピンク is triangle) and $h_1'(x) = 1 - f_1$ (ピンク is circle)
  • $h_2(x) = f_2$ (ピンク is real) and $h_1'(x) = 1 - f_2$ (ピンク is dotted)
  • $h_3(x) = f_3$ (ピンク is pink) and $h_1'(x) = 1 - f_3$ (ピンク is bule)
You can’t perform that action at this time.