How to set the epoch_size and max_epoch

[liujiqiang999]

Hi, Could you tell me how to set these two parameters? How long did it take to train in the paper ? Thank you.

[glample]

Hi,

This is really up to you how to set --epoch_size, it depends on how often you want to evaluate your model (not so often because evaluation takes a bit of time). I would recommend something like --epoch_size 500000.

Regarding --max_epoch, this is only implemented to prevent the model from running indefinitely. But usually, if you define a validation metric as --stopping_criterion bleu_en_fr_valid,10 (to kill the experiment if the BLEU on en -> fr has not improved over 10 epochs), the model will end up converging pretty quickly so the value of max_epoch is not relevant, and the best is just to set it to something very high so that it doesn't have any effect. Default value is 100000 and you can just let this.

[liujiqiang999]

Hi, normally epoch_size means the number of times all training set is trained once. We only use 5% training data for each epoch If using dataset and hyperparameters you provide. Is it wrong to understand?

[glample]

You are correct. Here, an epoch is not an iteration over the training set, it's an iteration over an arbitrary number of sentences. I do it this way because the training set is too big to evaluate only at the end of regular epochs.

[liujiqiang999]

Ok, thank you.