Does it work in not so deep architectures?

[wotulong]

Thanks for your greate job.Do you have any experiment of ReZero applied in different layers of transformers, like 1 layer Transformer layer and it performance , 2 layer Transformer layers and it performance, and so on.Does it make convergence faster in not so deep net?Thank you.

[calclavia]

In our paper, we experimented with Transformers with 12 layers and found speed increase. We have not tried shallower models, but we hypothesize it should also bring speed improvements.

[wotulong]

Thanks you.I'll try in my project later which is not so deep.

[bluesky314]

Any update @wotulong on shallower models?