should mBART-large-en-ro have decoder_start_token_id by default?

[sshleifer]

Hypothesis: since the argument prepend_bos is set to "False" in fairseq/examples/README.md, mbart-large-en-ro does not need decoder_start_token_id.

TODO:

• create branch that deletes decoder_start_token_id. Setting it to None in the config might not be enough.
• verify that decoder_start_token_id is in fact not being used by setting a breakpoint in generate.
• run_eval.py on wmt-en-ro/test and see if BLEU is >= 26.46, the score with decoder_start_token_id=250020.

[KMFODA]

Hi @sshleifer, I'd like to contribute and help out here if still needed. My thinking is to remove decoder_start_token_id from run_eval.py and generation_utils.py and change the following code:

transformers/src/transformers/generation_utils.py

Lines 403 to 409 in 6028ed9

	# create empty decoder_input_ids
	input_ids = torch.full(
	(effective_batch_size * num_beams, 1),
	decoder_start_token_id,
	dtype=torch.long,
	device=next(self.parameters()).device,
	)

to:

        input_ids = torch.full(
            (effective_batch_size * num_beams, 1),
            250020,
            dtype=torch.long,
            device=next(self.parameters()).device,
        )

[sshleifer]

I dont think that change will do anything since decoder_start_token_id = 250020.

What I would do is change the 250020 to a bos_token_id (0, I think) or a pad_token_id (1) and see what the BLEU score is.

[KMFODA]

Ah yes that makes sense. I tried those two and the eos_token_id and got the following results:

ID	BLEU Score
eos_token_id (2)	28.22
decoder_start_token_id (250020)	28.06
pad_token_id (1)	26.79
bos_token_id (0)	26.01

[sshleifer]

Super interesting, thanks for running that. It seems like I should change decoder_start_token_id in the mbart-large-en-ro config to 2. Do you have opinions on mbart-large-cc25?

[KMFODA]

No problem! Yes I think configuring decoder_start_token_id to 2 is a good idea. Unfortunately, I'm getting the same issues you're getting with mbart-large-cc25 (output's in English not Romanian and missing the first word when I use bos_token_id or 250020 and gibberish with eos/pad_token_id) and don't understand why that's the case. I'll investigate and post any useful findings.

[sshleifer]

I think I fixed this another way in #6526
on master

python run_eval.py facebook/mbart-large-en-ro $ENRO_DIR/test.source eos_baseline_enro_test_generations.txt \
--reference_path $ENRO_DIR/test.target \
--score_path baseline_test_bleu_eos.json --bs 32 --task translation --fp16

=> {'bleu': 26.81}

python run_eval.py facebook/mbart-large-en-ro $ENRO_DIR/test.source \
eos_baseline_enro_test_generations.txt --reference_path $ENRO_DIR/test.target \
--score_path baseline_test_bleu_eos.json --bs 32 --task translation --fp16  \
--decoder_start_token_id 2

{'bleu': 11.57} (and takes 40 mins!)

in the original fairseq I get 26.83.

[sshleifer]

Gunna close this since the score is now basically the same as fairseq. Thanks for your help!