The difference Rouge-L between the pyrouge package and files2rouge

[kugwzk]

Hi, @pltrdy Thx for this nice rouge package～
But I run files2rouge got an error like:
Can't locate LWP/UserAgent.pm in @inc (@inc contains: /users/XXX/.files2rouge /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .)

/users/XXX/.files2rouge is my files2rouge setup path during the installation.
In fact, I install LWP/UserAgent.pm in my perl5 path, but I am not a root user so I can't install any perl package in /usr path.
Therefore, I want to know whether there is a way to set the perl path for rouge-1.5.5.pl?

[kugwzk]

Fixed. I can directly set the path in the first line of the install ROUGE-1.5.5.pl.
BTW, I found a similar problem like #43 . I follow the Readme of BART for reproducing CNN/DM results.
I directly use the public checkpoint for CNN/DM, so maybe there is no problem in training/inference.
I use the Corenlp to tokenized the reference and prediction file.
I use the pyrouge or files2rouge to evaluate.
The pyrouge results are below:
`1 ROUGE-1 Average_R: 0.51297 (95%-conf.int. 0.51018 - 0.51576)
1 ROUGE-1 Average_P: 0.40694 (95%-conf.int. 0.40444 - 0.40952)
1 ROUGE-1 Average_F: 0.44288 (95%-conf.int. 0.44063 - 0.44507)

1 ROUGE-2 Average_R: 0.24449 (95%-conf.int. 0.24151 - 0.24725)
1 ROUGE-2 Average_P: 0.19458 (95%-conf.int. 0.19220 - 0.19700)
1 ROUGE-2 Average_F: 0.21127 (95%-conf.int. 0.20876 - 0.21368)

1 ROUGE-L Average_R: 0.35718 (95%-conf.int. 0.35430 - 0.35986)
1 ROUGE-L Average_P: 0.28160 (95%-conf.int. 0.27928 - 0.28400)
1 ROUGE-L Average_F: 0.30727 (95%-conf.int. 0.30490 - 0.30957)`

This R-L result is very lower the expected 40.90.
The files2rouge results are below:

1 ROUGE-1 Average_R: 0.40390 (95%-conf.int. 0.40134 - 0.40648)
1 ROUGE-1 Average_P: 0.50956 (95%-conf.int. 0.50689 - 0.51227)
1 ROUGE-1 Average_F: 0.43971 (95%-conf.int. 0.43750 - 0.44215)

1 ROUGE-2 Average_R: 0.19393 (95%-conf.int. 0.19147 - 0.19646)
1 ROUGE-2 Average_P: 0.24389 (95%-conf.int. 0.24102 - 0.24680)
1 ROUGE-2 Average_F: 0.21063 (95%-conf.int. 0.20813 - 0.21326)

1 ROUGE-L Average_R: 0.37440 (95%-conf.int. 0.37190 - 0.37693)
1 ROUGE-L Average_P: 0.47246 (95%-conf.int. 0.46979 - 0.47529)
1 ROUGE-L Average_F: 0.40764 (95%-conf.int. 0.40535 - 0.41010)

We can observe the difference on R-L. Could you give some advice for this situation?

[pltrdy]

I don't know how pyrouge works but if I remember right we calculate sentence level ROUGE-L (check original article for reference). Maybe pyrouge does not split sentence correctly, thus giving different results.

[kugwzk]

Thanks a lot.