Tow confused parts

[MaXXXXfeng]

1. From the code trainer.py(function-test) & utils.py(function-test_rouge), I think you compare cnndm_step50000.candidate and cnndm_step50000.gold to compute rouge for model evaluation. In my comprehension, cnndm_step50000.gold is oracle summary, generated from greedy algorithms, that is, it is not the abstractive summary from original document. I wonder why you take cnndm_step50000.gold as ref, but not the abstractive summary of the document? I think taking original abstractive summary as ref will give more comparable rouge score.

2.In the paper's Table1: Test set results on the CNN/DailyMail dataset using ROUGE F1, you show rouge score of Oracle and other model. I want to know how do you calculate Oracle_ROUGE-1(52.59; 31.24; 48.87), and taking what as ref? And how do you calculate BERTSUM+Transformer_ROUGE-1(43.25; 20.24; 39.63), and taking what as ref?

[nlpyang]

No, .gold is the highlights, not the oracle