Experiment with constant learning rate with transformers

[cwmeijer]

Do a couple of runs with constant learning, varied systematically between runs, and find out if we can find some learning rate that consistently reduces loss. After that we can experiment with more complicated schemes.

[egpbos]

I started a run, effortless-cosmos-19, with the cyclic scheduler with maximum and minimum set to values that may be more sane for this model (based on this article). So far, it is at least outperforming all previous runs. Will let it run until Monday 8 a.m. (according to my estimation) and then we'll see how it went.

Already, now, one interesting feature is the correlation of the periodic pattern in the loss with the learning rate. This might suggest that, as also mentioned in the article, some kind of decreasing maximum might be helpful to press down the big (seemingly useless) peaks in the loss and make it converge even faster.

[egpbos]

New run dainty-dawn-20 also logs validation loss on wandb.

[egpbos]

I think we can close this, since the training loss is very effectively reduced now. Next problem is validation loss, issue #30.