-
Notifications
You must be signed in to change notification settings - Fork 228
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Seq2Seq Metrics QOL: Bleu, Rouge #107
Comments
Hi ! Thanks for letting us know your experience :) |
So what is the right way to add a batch to compute BLEU? |
prediction = [['Hey', 'how', 'are', 'you', '?']] also tried this kind of things lol |
Hi ! As described in the documentation for
Therefore you can use this metric this way: from datasets import load_metric
predictions = [
["hello", "there", "general", "kenobi"], # tokenized prediction of the first sample
["foo", "bar", "foobar"] # tokenized prediction of the second sample
]
references = [
[["hello", "there", "general", "kenobi"], ["hello", "there", "!"]], # tokenized references for the first sample (2 references)
[["foo", "bar", "foobar"]] # tokenized references for the second sample (1 reference)
]
bleu = load_metric("bleu")
bleu.compute(predictions=predictions, references=references)
# Or you can also add batches before calling compute()
# bleu.add_batch(predictions=predictions, references=references)
# bleu.compute() Hope this helps :) |
Putting all my QOL issues here, idt I will have time to propose fixes, but I didn't want these to be lost, in case they are useful. I tried using
rouge
andbleu
for the first time and wrote down everything I didn't immediately understand:calculate_rouge
andcalculate_bleu
functions here: https://github.com/huggingface/transformers/blob/master/examples/seq2seq/utils.py#L61What I tried
Rouge experience:
BLEU experience:
All of these raise
ValueError: Got a string but expected a list instead: 'im daniel'
Doc Typo
This says
dataset=load_metric(...)
which seems wrong, will causeNameError
cc @lhoestq, feel free to ignore.
The text was updated successfully, but these errors were encountered: