Skip to content

Commit 545eee7

Browse files
committed
Attempt to stop people from publishing non-comparable BLEU scores, as discussed in statmt meeting
1 parent eced95d commit 545eee7

File tree

1 file changed

+3
-0
lines changed

1 file changed

+3
-0
lines changed

scripts/generic/multi-bleu.perl

+3
Original file line numberDiff line numberDiff line change
@@ -168,6 +168,9 @@ sub add_to_ref {
168168
$length_translation,
169169
$length_reference;
170170

171+
172+
print STDERR "Do not publish scores from multi-bleu.perl. The scores depend on your tokenizer, which is unlikely to be reproducible from your paper or consistent across research groups. Instead you should detokenize then use mteval-v14.pl, which has a standard tokenization. Scores from multi-bleu.perl can still be used for internal purposes when you have a consistent tokenizer.\n";
173+
171174
sub my_log {
172175
return -9999999999 unless $_[0];
173176
return log($_[0]);

0 commit comments

Comments
 (0)