discuss the solutions to Not fully recovering spikes

[pengzhangzhi]

Not fully recovering spikes are quite common in model training at scale. Like what I have:

It's worth discussing the causes and potential solutions to that.
My general guesses are:

• some of the toxic data points at certain iterations
• The learning rate.

Welcome everyone to talk about your experiences dealing with the spikes...

Reference:

https://github.com/stas00/ml-engineering/blob/master/training/instabilities/training-loss-patterns.md

[stas00]

Am I understanding that you'd like to keep this issue open for some time and invite a community discussion, correct?

---

Usually in this kind of situation we would roll back to a checkpoint right before the spike (but some 10-100 steps back) and then either try a new data (if bad data was the trigger) or lower the lr. And observe the grad norm while you're at it, since most likely your grads are spikes.

Publicly available training LLM/VLM logbooks is a good source of learning of how such problems were dealt with by others.

[pengzhangzhi]

yeah! Or close it when we have systematic solutions/ debugging guidances to this problem. I feel like the issue section is a great place for ppl to talk about their experiences dealing with model training at scale.

Thanks for the guidance! Will share my experiences later :)

[pengzhangzhi]

My solutions are:

1. increase the batch size
2. reduce the lr

However, I think the problem is somewhere else. I am sampling training data based on sequence length, i.e., a batch of sequences with similar lengths are sampled together. Thus the reason why the loss spikes is it switches to train on longer sequence batches.

[pengzhangzhi]

Hi @stas00 A random question. What is your principle for tuning the learning rate? The larger the better? My intuition is using the largest lr to make the learning phrase converge faster.

[stas00]

@pengzhangzhi, it is probably better if we use the github Discussion feature for what you intend and leave Issues for what they are.

I have just enabled it here https://github.com/stas00/ml-engineering/discussions - so you can start a discussion there moving your comments so far and then we can let other people know that they can join and share their insights?

What do you think?

I have never used that feature so hopefully it's easy to use - please let me know if you run into any issues

[pengzhangzhi]

Thanks!! @stas00

[stas00]

but you're starting a discussion, yes? We can discuss your questions there, then and invite others to contribute. That'd be much more productive than just you and I talking.