Bug in sentence level BLEU comparison

[madaan]

Description

The report

N sentences where Sys A > Sys B at sentence-level BLEU

will generate wrong output if:

a) Sys A never generates sentences that have a higher BLEU score

b) There are less than N sentences in the set of sentences to be analyzed

Screenshots

Files

compare-mt/compare_mt/reporters.py

Line 507 in 41700d6

print(f'--- {report_length} sentences where {sright}>{sleft} at {self.scorer.name()}')

To Reproduce

Use the SysA, SysB and Ref outputs located at https://gist.github.com/madaan/2cec36a7b18dfeea3904ddfff1e19312 and run compare-mt with the default options.

Tasks

• Add checks before emitting the report at:

  compare-mt/compare_mt/reporters.py

  Line 507 in 41700d6

  print(f'--- {report_length} sentences where {sright}>{sleft} at {self.scorer.name()}')

I can take a stab at it if you guys think this should be fixed.

Thanks!

[neubig]

Thanks a lot! I think it's probably better to just change the message however, from `{sright}>{sleft}' to something indicating that this is just the maximum difference (which could also be negative). PR would be welcome.