Standardized modelgauge column names #50
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
| except Exception as e: | ||
| print(f"Failed to log stats for {annotator_uid}: {e}") | ||
| with AnnotationDataset(data_path, "r") as dataset: | ||
| for item in dataset: |
There was a problem hiding this comment.
I'm not totally sure how this works when there are multiple annotators? If I'm reading the __iter__ in modelgauge.dataset correctly, each row produces one item, but I think if there are multiple annotators, each row will contain multiple annotator_uids? Or did that change too in the modelgauge PR?
There was a problem hiding this comment.
That changed in the modelgauge PR! Every row is one response and one annotation.
There was a problem hiding this comment.
That's probably why the randomness changed! Since before it was looping row, and then a for loop around the annotators per row, but the order of the annotators may not match the order modelgauge is now producing.
I'm good on this then!
No description provided.