Falcon Loss Not Decreasing During Training

[LamOne1]

I'm using pretrain code with falcon 7B. I've noticed that the loss didn't change for 400 iterations.

iter 1: loss 11.0666, time: 13381.00ms, speed: 306 toks/s/device
....
iter 400: loss 11.0666, time: 19090.34ms, speed: 214 toks/s/device

[carmocca]

Can you share exactly what script you ran?

[LamOne1]

I was using redpajama pretrain script as it is.
However I prepared my data with PackedDataset type np.uint16 instead of np.int32, could be the problem?

[carmocca]

I don't think so. But you might need to tweak hyperparameters. This is the dark art of machine learning 😉