You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
When I define a Book to support multiple raters all raters can give a judgement for a query-doc pair. In the selection strategy it clearly states "up to three", so I report this as a bug.
To Reproduce
Steps to reproduce the behavior:
In a Quepid case create a book with the Selection strategy "Multiple Raters"
Make sure to share the book with a team that consists of four or more persons/accounts
Populate the book with query/doc pairs
To restrict the number of judgements you can edit the book and cut off everything after position 1
Make all judges collect some judgements
Once every judge has rated all query/doc pairs you will see that query/doc pairs are not restricted to three judgements.
Expected behavior
Only select query/doc pairs to be judged when this query/doc pair has less than three ratings.
Screenshots
n/a
Additional context
n/a
The text was updated successfully, but these errors were encountered:
Describe the bug
When I define a Book to support multiple raters all raters can give a judgement for a query-doc pair. In the selection strategy it clearly states "up to three", so I report this as a bug.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Only select query/doc pairs to be judged when this query/doc pair has less than three ratings.
Screenshots
n/a
Additional context
n/a
The text was updated successfully, but these errors were encountered: