Training from scratch, initialisation recommendations

[prajwalkr]

Hi

I am a Ph.D. student, trying to train a part of Whisper from scratch. Could you please let me know a few training details, such as weight initialization and the optimizer learning rate schedules? The weight initialization mentioned in the paper is: Gaussian Fan-In, can you explain (or link) to what this exactly is?

[jayarjo]

Have you ever figured this out?