MACE for Multi-label case?

[shahhaard47]

Hello, thanks for the library!

I have a scenario where each rater can apply multiple labels to each example. I have 20 labels and 4 raters.

I'm new to this and I would really appreciate if you could suggest some way for to apply MACE to get rater competence as well as prediction of most likely label.

[dirkhovy]

Thanks! Yes, that sounds like it is exactly a job for MACE (as long as you have enough items). Running MACE will give you both a prediction file as well as one with the competence of each annotator. Dirk Hovy Web: www.dirkhovy.com Skype: dirkhovy Twitter: @dirk_hovy

…

On 5 Dec 2020, at 03:53, Haard ***@***.***> wrote: Hello, thanks for the library! I have a scenario where each rater can apply multiple labels to each example. I have 20 labels and 4 raters. I'm new to this and I would really appreciate if you could suggest some way for to apply MACE to get rater competence as well as prediction of most likely label. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABK7AXTEEY6UMH3OSWAHA23STGOEPANCNFSM4UOG5XAA>.

[shahhaard47]

Hello, thank you so much for your response. I'm sorry for not stating question more clearly. What I meant by "multi-label" is, each sample could have more than one label applied by each rater. So for sample x_i, rater A can apply 2 labels, rater B can apply 3 labels, rater C can apply 1 label, and so on.

I see that the csv files only allow one label per sample per rater. To elaborate further, the README states:

each line represents an item, and each column represents an annotator

However, each annotator can have multiple labels for each item. How do I apply MACE to this use case? If I can't, could you give any pointers on how I could adapt it maybe?

[dirkhovy]

Ah, ok, I see: I casually read multi-LABEL as multi-CLASS (the classic mistake). MACE is not natively set up to deal with multiple labels per item, but there is a relatively simple way to address this: you could make one copy of the input file per class and simply use binary indicators for each annotator if they chose that class. So, if your first item looks like this: Annotator 1: A, C Annotator 2: A, B, D Annotator 3: Annotator 4: B, C You would make an input file for label A that looks like this: 1,1,,0 And for label B: 0,1,,1 Cheers, Dirk Dirk Hovy Web: www.dirkhovy.com Skype: dirkhovy Twitter: @dirk_hovy

…

On 15 Dec 2020, at 07:26, Haard ***@***.***> wrote: Hello, thank you so much for your response. I'm sorry for not stating question more clearly. What I meant by "multi-label" is, each sample could have more than one label applied by each rater. So for sample x_i, rater A can apply 2 labels, rater B can apply 3 labels, rater C can apply 1 label, and so on. I see that the csv files only allow one label per sample per rater. To elaborate further, the README states: each line represents an item, and each column represents an annotator However, each annotator can have multiple labels for each item. How do I apply MACE to this use case? If I can't, could you give any pointers on how I could adapt it maybe? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABK7AXXP2ANTLIM4P36QNK3SU36RDANCNFSM4UOG5XAA>.