50 constant adam

[cwmeijer]

add constant learning rate scheduler.
constant_lr argument is only used in transformer experiment for now.

[bhigy]

Maybe we should take this as an opportunity to use a uniform scheduler creation procedure. I see that different experiment functions use different code (e.g. asr.experiment() has very different rules than basic.experiment()). This should include:

• a uniform set of accepted optimizers (adadelta, adam...)
• a uniform set of scheduling options (none, cyclic, noam; where I would use none instead of constant as adaptive optimizers don't use a constant learning rate even without scheduler)
• uniform parameter names (constant_lr -> lr)
• a single function in scheduler.py to handle this, called from all experiment() functions

[cwmeijer]

That sounds like an excellent idea. I propose we create a new issue in order to fix the things you describe given the number and impact of the proposed changes.