[BUG]: (Update) Together AI fine-tuning always stops at step 10 ❌ Need further explanations on how steps work in Together AI ✅

[ruiyiw]

Description of the bug

All fine-tuning processes still stop at step 10. According to @lwaekfjlk, no early stopping code is found in the source code. And the curves shown in the below image stop at different losses. Need further experiments.

Steps To Reproduce

No response

Additional Information

Here is a comparison between two different fine-tuning settings:
epoch = 2, batch=4 -> epoch 1 stopped at step 5, epoch 2 stopped at step 10
epoch = 4, batch=32 -> epoch 1 stopped at step 3, epoch 2 stopped at step 6, epoch 3 stopped at step 9, epoch 4 stopped at step 10.

[ruiyiw]

Total step number changes in a new fine-tuning setting:
llama-2-13b-chat w/ 587 datapoints ->
epoch = 2, batch = 8, lr=5e-5

[ruiyiw]

Need further explanations on how the steps work in Together AI

[lwaekfjlk]

@ruiyiw Even though we set the learning rate as 5e-5, the peak point of the learning rate is actually 1e-5? Is that expected?

[ruiyiw]

Yes. But why does the lr start from 0 and gradually reach 1e-5?

[lwaekfjlk]

Should be included in #14