local attention with unidirectional lstm not converging

[manish-kumar-garg]

Hi @albertz ,
I tried with different learning rates but model seems to be not converging after changing bidirectional lstm to unidirectional lstm in case of local attention setup.
Can you suggest something?
What else should I try.

[albertz]

It seems a very related question was also asked in #42.
I have not tried unidir LSTMs in the encoder yet, so I don't know. You probably should play around with all the available hyper params, e.g.:

• Learning rate (initial at warmup, and highest learning rate after warmup)
• Learning rate warmup length (num epochs)
• Pretraining. E.g. start number of layers (try 2). Initial time reduction (try increasing it, e.g. 6, 8, 16, or even 32). Try make it longer (more repetitions). Etc.
• No or less SpecAugment in the beginning.
• Higher batch size in the beginning. Or gradient accumulation in the beginning.
• Curriculum learning, i.e. the epoch_wise_filter option.
• ...

Let this smallest network with highest time reduction, high batch size, less/no SpecAugment etc train like that for as long as needed, before increasing anything. This small network should first get some half-way good score. Only when you see that, at that point the pretraining can increase the depth and other things, but only carefully and slowly (such that the network does not totally break again).

[manish-kumar-garg]

Thanks @albertz for suggesting these.
I trained following models upto pretraining epochs (45) with following observation of loss:

1. Base model - asr_2018_attention - below hyperparameters:

pretrain = {"repetitions": 5, "construction_algo": custom_construction_algo}
learning_rate = 0.0008
learning_rates = list(numpy.linspace(0.0003, learning_rate, num=15))  # warmup

2. Uni lstm size 1024 with all hyperparameters same as 1:
   

3. Uni lstm size 1024 with all hyperparams same as 1 except:
   pretrain = {"repetitions": 7, "construction_algo": custom_construction_algo}
   

4. Uni lstm size 1024 with all hyperparams same as 1 except:
   learning_rate = 0.0005
   

5. Uni lstm size 1024 with all hyperparams same as 1 except warmup steps 10:
   learning_rates = list(numpy.linspace(0.0003, learning_rate, num=10)) # warmup
   

6. Uni lstm size 1024 with all hyperparams same as 1 except warmup steps 20:
   learning_rates = list(numpy.linspace(0.0003, learning_rate, num=20)) # warmup
   In this case loss becomes nan after 10 epochs

7. Uni lstm size 1536 with all hyperparameters same as 1:
   

All the models are with global attention

[manish-kumar-garg]

Seems like decreasing the learning rate helps.
Also, increasing the lstm cell size to 1536 helps a bit, however, not much.

What other combination do you suggest to try next?

[albertz]

All the things which I wrote already (here), but basically everything else as well.

[manish-kumar-garg]

Lowering the learning rate lr=0.0005 and lr_init=0.0002 worked for me.
Thanks!