-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MACE for Multi-label case? #1
Comments
Thanks! Yes, that sounds like it is exactly a job for MACE (as long as you have enough items).
Running MACE will give you both a prediction file as well as one with the competence of each annotator.
Dirk Hovy
Web: www.dirkhovy.com
Skype: dirkhovy
Twitter: @dirk_hovy
… On 5 Dec 2020, at 03:53, Haard ***@***.***> wrote:
Hello, thanks for the library!
I have a scenario where each rater can apply multiple labels to each example. I have 20 labels and 4 raters.
I'm new to this and I would really appreciate if you could suggest some way for to apply MACE to get rater competence as well as prediction of most likely label.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub <#1>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABK7AXTEEY6UMH3OSWAHA23STGOEPANCNFSM4UOG5XAA>.
|
Hello, thank you so much for your response. I'm sorry for not stating question more clearly. What I meant by "multi-label" is, each sample could have more than one label applied by each rater. So for sample x_i, rater A can apply 2 labels, rater B can apply 3 labels, rater C can apply 1 label, and so on. I see that the csv files only allow one label per sample per rater. To elaborate further, the README states:
However, each annotator can have multiple labels for each item. How do I apply MACE to this use case? If I can't, could you give any pointers on how I could adapt it maybe? |
Ah, ok, I see: I casually read multi-LABEL as multi-CLASS (the classic mistake).
MACE is not natively set up to deal with multiple labels per item, but there is a relatively simple way to address this: you could make one copy of the input file per class and simply use binary indicators for each annotator if they chose that class.
So, if your first item looks like this:
Annotator 1: A, C
Annotator 2: A, B, D
Annotator 3:
Annotator 4: B, C
You would make an input file for label A that looks like this:
1,1,,0
And for label B:
0,1,,1
Cheers,
Dirk
Dirk Hovy
Web: www.dirkhovy.com
Skype: dirkhovy
Twitter: @dirk_hovy
… On 15 Dec 2020, at 07:26, Haard ***@***.***> wrote:
Hello, thank you so much for your response. I'm sorry for not stating question more clearly. What I meant by "multi-label" is, each sample could have more than one label applied by each rater. So for sample x_i, rater A can apply 2 labels, rater B can apply 3 labels, rater C can apply 1 label, and so on.
I see that the csv files only allow one label per sample per rater. To elaborate further, the README states:
each line represents an item, and each column represents an annotator
However, each annotator can have multiple labels for each item. How do I apply MACE to this use case? If I can't, could you give any pointers on how I could adapt it maybe?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABK7AXXP2ANTLIM4P36QNK3SU36RDANCNFSM4UOG5XAA>.
|
Hello, thanks for the library!
I have a scenario where each rater can apply multiple labels to each example. I have 20 labels and 4 raters.
I'm new to this and I would really appreciate if you could suggest some way for to apply MACE to get rater competence as well as prediction of most likely label.
The text was updated successfully, but these errors were encountered: