add BartConfig.force_bos_token_to_be_generated

[sshleifer]

This PR adds a config flag that makes a generation hack from bart's adjust_logits_during_generation, optional. This change (setting the flag to False) improves metrics for bart-large-xsum and mbart-large-en-ro, but not bart-large-cnn (hence the need for the flag).

I remember @patrickvonplaten asked me about this in February, and I ran on CNN and then decided the hack needed to stay, could have been more careful.
cc @patil-suraj

Todo

[x] test xsum
[x] test cnn
[x] test en-ro
[x] update cnn config
[x] test pegasus
[x] update distilbart configs
[] should max_length be changed in xsum config?

TODO:

• check pegasus
• should max_length be 1 lower for configs where hack is removed? Yes.

Metrics

(all on val.source vs. val.target) the first number is runtime over the whole val set. "noforce" means this PR without any config change. (so BOS token is not forced to be generated.)

bart-large-cnn
master: 86:23 {"rouge1": 44.79, "rouge2": 21.64, "rougeL": 31.18}
noforce: 87:40 {'rouge1': 44.26, 'rouge2': 21.22, 'rougeL': 30.72}

bart-large-xsum
master: 41:59, {'rouge1': 45.16, 'rouge2': 21.77, 'rougeL': 36.35}
noforce: 34:12, {'rouge1': 45.45, 'rouge2': 22.38, 'rougeL': 37.25}

mbart-large-en-ro
master: 04:58 BLEU=27.83
noforce: 04:42, BLEU=28.15

pegasus-xsum
master: 56:12 {'rouge1': 46.69, 'rouge2': 24.13, 'rougeL': 38.79}
noforce: 54:15 {'rouge1': 46.98, 'rouge2': 24.43, 'rougeL': 39.11}

Commands

export DATA_DIR=wmt_en_ro
python run_eval.py facebook/mbart-large-en-ro \
    $DATA_DIR/val.source gens/mbart-enro-branch-gens.txt \
    --reference_path $DATA_DIR/val.target \
    --score_path gens/mbart-enro-master-bleu.json \
    --task translation_en_to_ro \
    --device cuda \
    --fp16 \
    --bs 32

export DATA_DIR=$CNN_DIR
python run_eval.py facebook/bart-large-cnn \
    $DATA_DIR/val.source gens/cnn_val_generations_no_force.txt \
    --reference_path $DATA_DIR/val.target \
    --score_path gens/cnn_gen_no_force_rouge.txt \
    --device cuda \
    --fp16 \
    --bs 32


export DATA_DIR=$CNN_DIR
python run_eval.py facebook/bart-large-cnn \
    $DATA_DIR/val.source gens/cnn_val_generations_master.txt \
    --reference_path $DATA_DIR/val.target \
    --score_path gens/cnn_gen_master_rouge.txt \
    --device cuda \
    --fp16 \
    --bs 32

[codecov]

Codecov Report

Merging #6526 into master will decrease coverage by 2.39%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #6526      +/-   ##
==========================================
- Coverage   80.59%   78.19%   -2.40%     
==========================================
  Files         156      156              
  Lines       28058    28055       -3     
==========================================
- Hits        22612    21939     -673     
- Misses       5446     6116     +670     

Impacted Files	Coverage Δ
src/transformers/configuration_bart.py	94.00% <100.00%> (+0.12%)	⬆️
src/transformers/modeling_bart.py	95.56% <100.00%> (-0.20%)	⬇️
src/transformers/modeling_marian.py	90.00% <100.00%> (ø)
src/transformers/optimization.py	25.55% <0.00%> (-70.00%)	⬇️
src/transformers/pipelines.py	26.26% <0.00%> (-53.43%)	⬇️
src/transformers/modeling_tf_bert.py	66.00% <0.00%> (-32.38%)	⬇️
src/transformers/optimization_tf.py	33.33% <0.00%> (-24.33%)	⬇️
src/transformers/modeling_tf_auto.py	48.79% <0.00%> (-18.08%)	⬇️
src/transformers/data/processors/squad.py	13.76% <0.00%> (-14.38%)	⬇️
src/transformers/modeling_auto.py	64.36% <0.00%> (-14.37%)	⬇️
... and 12 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2060181...5e951e1. Read the comment docs.

[sshleifer]

just cleanup

[sshleifer]

summaries change a bit for xsum, but ROUGE increases.

[sgugger]

LGTM, thanks for cleaning this. Just a few nits on the doc.

[patrickvonplaten]

LGTM!