Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate classifiers against the labeled ambiguous data #4

Closed
martinthenext opened this issue Apr 2, 2014 · 1 comment
Closed

Validate classifiers against the labeled ambiguous data #4

martinthenext opened this issue Apr 2, 2014 · 1 comment

Comments

@martinthenext
Copy link
Owner

Implement a procedure to measure agreement of a classifier with the labeled ambiguous data. For that purpose:

  1. Annotation class should be modified to store ambiguous data as well
  2. Function should be implemented in data.py to deserialize labeled ambiguous data (from MTurk or expert) to a list of Annotation objects.
  3. Module should be implemented to test classifiers against this data - analogous to cv.py
@martinthenext
Copy link
Owner Author

Input data format for labeled data can be viewed here:

https://kitt.cl.uzh.ch/kitt/mantracrowd/disambig/vote_results.csv?AgreementThr=0.6

where AgreementThr designates agreement threshold

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant