adafactor vs adam

[nicolabertoldi]

Description

I am interested in using adafactor (instead of adam) because it allows checkpoints of smaller size and according to this paper achieves also good performance wrt to adam.
But according to the logs it seems that the bleu is much lower, as you can see below.

Is there any setting (different from default) which is specific for Adafactor?

approx_bleu for Adafactor: (one evaluation every 5K steps)
5K: 0.057251852
10K: 0.16007528
15K: 0.25117296
20K: 0.29413137
25K: 0.3245068
30K: 0.3451813
35K: 0.366105

approx_bleu for Adam: (one evaluation every 5K steps)
5K: 0.32362464
10K: 0.4280433
15K: 0.47145975
20K: 0.4960477
25K: 0.51056355
30K: 0.52096725
35K: 0.53078866
40K: 0.5337893
45K: 0.5363042
50K: 0.53831

Environment information

OS: Ubuntu 16.04

$ pip freeze | grep tensor
tensor2tensor==1.6.3
tensorboard==1.8.0
tensorflow-gpu==1.8.0


$ python -V
Python 2.7.12

For bugs: reproduction and error logs

# Steps to reproduce:
t2t-datagen --data_dir t2t_data/datagen --tmp_dir ./t2t_data/tmp --problem translate_enfr_wmt_small8k

t2t-trainer --data_dir t2t_data/datagen --tmp_dir ./t2t_data/tmp --problem translate_enfr_wmt_small8k --model transformer --hparams_set transformer_base --output_dir ./t2t_data/model_adafactor --local_eval_frequency=500 --train_steps=1000 --worker_gpu=1 --hparams batch_size=3072,optimizer=Adafactor

t2t-trainer --data_dir t2t_data/datagen --tmp_dir ./t2t_data/tmp --problem translate_enfr_wmt_small8k --model transformer --hparams_set transformer_base --output_dir ./t2t_data/model_adam --local_eval_frequency=500 --train_steps=1000 --worker_gpu=1 --hparams batch_size=3072

[martinpopel]

I have good experience with Adafactor and NMT with the default hparams in T2T. I have no experience with *small8k. Have you tried 32k vocab?

[nicolabertoldi]

What about the convergence speed. Is it faster or slower than Adam?
Could you please give a look at my hparams.txt and tell me if there is any "wrong" parameter?

For sure I will try with 32K, but I expected that also with 8K the performance of Adafactor and Adam were similar.

[martinpopel]

In my experience the convergence speed was about the same, but I could afford a larger batch size, which was beneficial in some cases.
I usually used --hparams="batch_size=2900,max_length=150,learning_rate_warmup_steps=8000,learning_rate_schedule=rsqrt_decay,optimizer=Adafactor"
following my paper.

[nicolabertoldi]

my setting is very similar

one difference is about learning_rate_schedule
you suggest “rsqrt_decay”
whereas the default sets “constantlinear_warmuprsqrt_decay*rsqrt_hidden_size”

Why did you choose your choice?