dropout is 0.0

[dipsivenkatesh]

The dropout of the GPT model in GPTConfig class is set to 0.0, this means there won't be any dropout when training though, correct?

[amar-jay]

Yes, by deafult

[muerghq]

Many recent researches on LLMs proved that it's ok to not do dropout in pertaining. But you normally want dropout in fine tuning to avoid overfitting.

[siddharthji07]

Yes, Absolutely core because dropout of 0.0 during training shows that there is no dropout during training