Reproducing bertscore

[moussaKam]

Hello,

Thank you for the great work,

I am trying to reproduce BertScore for Table 2, block 2 in you paper. Where you evaluate different metrics against one reference.
I assume that you are using the first reference in the list of references that you provide in the processed files.

So my code looks like that for example:

with open('data/processed/processed.summarization.cnndm') as f:
    data = json.load(f)
candidates = [data[el]['hypothesis'] for el in data]
references = [data[el]['references'][0] for el in data]

Then I'm running bertscore after saving the references and the candidates in files using:

bert-score -r references.txt -c candidates.txt -s --lang en -m bert-base-uncased --idf > bert_score.txt

I trained with\without --idfand with both bert-base-uncased and roberta-large.

In all cases I'm obtaining values different than those in the json file that I load as follows:

bertscore = [data[el]['ref_1']['bertscore_f1'] for el in data]

Can you tell me please what are the exact options that you use to compute bertscore?

[ThomasScialom]

Hi,

I used the default configuration in my metric reporter, see in beametrics/metrics/metrics_hugging_face.py.

You can see that the model_type is set to 'bert-base-multilingual-cased', allowing the use of the same exact model for all the (multilingual) datasets.

Hope it can help.

Thom

[moussaKam]

Great! Thank you!