Replies: 1 comment 2 replies
-
Hi @vahuja4, thanks for the question! Even if you had 3 annotators (or any constant number) annotating all the data, CROWDLAB will still be helpful in providing more robust consensus labels for your dataset. This is because in addition to the annotations from your annotators, CROWDLAB utilizes information (predictions) from a trained classifier to determine the best consensus label for each example. The CROWDLAB algorithm will automatically weigh the trustworthiness of the annotators and classifier and ensemble their annotations/predictions to provide you a consensus label, alongside a score for each example indicating the confidence level of that consensus label. This process is similar regardless if you have a constant or varying number of annotators for each example. In fact, Appendix C of the CROWDLAB paper showcases CROWDLAB in use with a dataset where each example was annotated by 50 annotators! Lastly, you can use any trained classifier with CROWDLAB, but as with most of the cleanlab package, a better classifier would likely yield better results 🙂 |
Beta Was this translation helpful? Give feedback.
-
From the crowdlab paper: "Illustrating how many real-world multiannotator datasets look, Figure 1 shows a disparity in annotator quality as well as many examples whose consensus
label will be incorrect if we rely on majority vote (nonetheless often done in practice due to its straightforward appeal).
Unsurprisingly, consensus labels are more likely to be incorrect for those examples with fewer annotations. An effective
method to estimate consensus label quality should properly
account for the number of annotations an example has received, as well as the quality of the annotators who selected
these labels".
So, if we have 3 annotators annotating all the data, does the crowdlab algorithm still help with creating the consensus label as opposed to majority vote? Because there is no concept of fewer annotations in my dataset.
Beta Was this translation helpful? Give feedback.
All reactions