Skip to content

Commit

Permalink
Fix Brier Score (#1847)
Browse files Browse the repository at this point in the history
`gold_one_hot` needs to follow the dimension of predictions so that it still works when `--limit` is used and the indexes in gold does not cover all gold indexes.
  • Loading branch information
lintangsutawika committed May 24, 2024
1 parent 5f3a662 commit 7d747ea
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions lm_eval/api/metrics.py
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,10 @@ def ter(items):
@register_aggregation("brier_score")
def brier_score(items): # This is a passthrough function
gold, predictions = list(zip(*items))
bs, num_class = np.array(predictions).shape

gold = list(gold)
gold_one_hot = np.eye(np.max(gold) + 1)[gold]
predictions = list(zip(*items))[1]
gold_one_hot = np.eye(num_class)[gold]
return np.mean(np.sum((predictions - gold_one_hot) ** 2, axis=1))


Expand Down

0 comments on commit 7d747ea

Please sign in to comment.