Reducing BLEU score

[karthikpuranik11]

I tried to finetune the indicTrans model on 1603080 en-hi sentences from WAT 2021. Initially, I trained it for 3 epochs for 9 hours 20 minutes on GPU and got a BLEU score of 37.1. Then, I tried to continue the training the next day by restoring the last checkpoint which ran for 6 epochs for a period of 19 hours odd and finally gave a BLEU score of 36.2. None of the epochs on the second day seemed to produce the best checkpoint and the loss remained around 3.1 for the entire period.
What seems to be the problem and how I can solve it?

[gowtham1997]

We had similar observations when finetuning.

The fine-tuning works well for few epochs and after that, the model starts overfitting. So generally 2-3 epochs seem to be enough but make sure the learning rate is smaller (in range of 1e5, we tried 3e-5 and that worked for some datasets)

^ one of the loss plots when I tried fine-tuning with another dataset recently. You can see that the val loss goes down for 2 epochs and then it stars overfitting

You can try maybe reducing the learning rates or changing some hyperparameters and let us know if you manage to get better results.

[karthikpuranik11]

Thank you so much. It worked and I managed to get better BLEU scores.

[gowtham1997]

@karthikpuranik11
Good to know that you have better bleu scores now.

Did different hyperparameters help with better bleu scores after fine-tuning with en-hi WAT data or what did you do differently?

[karthikpuranik11]

I reduced the learning rates as suggested and also gave the max-epochs to be 3.

I was just curious to know how else could I detect overfitting from the training log file other than the loss?

[gowtham1997]

I guess you are referring to the fairseq training logs.

^ you can just compare the current validation loss to the best_val_loss and see how it changes. If train loss is going down and validation loss isn't improving or getting worse, you can stop the model (also adding early stopping with some patience)

In addition, you can also use wandb or tensorboard to track the losses like this:

fairseq-train ../dataset/final_bin \
...                      # other args like max_updates, learning rate etc
...
--wandb-project <wandb project name> (or) --tensorboard-logdir <path to store tensorboard logs>