specAugment policy and schedules

[akshatdewan]

Hi,

I wanted to run an experiment with LD augmentation policy (as described in the Google Brain paper ) along with D learning schedule.

I was wondering what would be the right way of doing something like with base2.conv2l.specaug.curric3.config.

I was thinking of doing:

1. Two additional masks in transform function just by simply calling random_mask two more times.
2. Slowing down the warm-up by increasing num from 10 to 20 or 40
3. Reducing the exponential LR decay newbob_learning_rate_decay from 0.9 to 0.95

Would it be a reasonable thing to do?

Thanks

[albertz]

We already have variations of that, where we also play around with scheduling of SpecAugment.
E.g. see Switchboard base2.conv2l.specaug4a.
@papar22 and @ZhouW321 also have some more variations, which we will upload soon to the repo.

Note that the random_mask in that config already runs multiple times (which is stochastically sampled). That are the options min_num and max_num. If you want that the mask is always runs exactly 3 times, just set min_num=3, max_num=3.

Yes sure, you can play around with learning rate warm-up as well. My experience however is that increasing usually will not help.

Reducing the LR decay helps when you want to increase your overall training time, i.e. train more epochs. And training longer usually helps. When you look at this original SpecAugment paper, you will see that they effectively train much longer than we do.

[akshatdewan]

Thanks for your answer, Albert!

I am sorry for a possibly naive question, but in the config example you mention above, the newbob_learning_rate_decay is 0.7.

My understanding is: LR_epoch_t+1=decay*LR_epoch_t. So if I am starting from a baseline model trained for 12.5 epochs using newbob_learning_rate_decay= 0.9, and I want to train another model for say 25 epochs, I should increase newbob_learning_rate_decay to say 0.95 instead of reducing it to 0.7, right?

[albertz]

Yes sure.