abnormal loss from train_sft.py

[houghtonweihu]

I tried train_sft.py with model = gpt2, but got the following warning:

fit colossalai - colossalai - WARNING: batch_id:2, abnormal loss: 10.97655200958252

What is the meaning of abnormal loss?

Thanks!

[binmakeswell]

Hi @houghtonweihu You can refer to the example loss. Obviously, your loss is too high.

[houghtonweihu]

Thanks @binmakeswell for your reply! Did you try the train_sft.py with model = gpt2? What I try to learn is: does train_sft.py also work for gpt2.