Variable number of labelers?

[fedor57]

Hi, I tried to test the algorithm over some real-life data. I found the limitation of having a fixed number of labelers a little bit impractical. I wonder if this is a particular implementation limitation or one of the aggregation model? What if a user provide some kind of reasonable upper limit for a task labelers count (3-5-7)? Thank you!

[sukrutrao]

Hi,
Thanks for your question. It is a limitation of the implementation, not of the model. We will update it to allow for a variable number of annotators per question.

[sukrutrao]

Hi,
This has been updated in #2. Now, setting k=0 or not setting k at all will make the algorithm use all available annotations for each data point.

Please re-open the issue if any problems persist. Thank you!

[fedor57]

Fantastic, thank you!!! Gonna test it and let you know!