use a smaller LR?

[152334H]

The Karparthy Constant used currently might be too high? The loss for this training run is not going down after LR increases beyond ~1e-4:

[tloen]

I've observed similar — updating to 2e-5 as in the original paper. I'll check in the morning if it works better.

[152334H]

this doesn't seem to actually help. Purple line is with 2e-5, green line 3e-4.

Maybe this is just the best result obtainable with the q_proj/v_proj parameters?

[tloen]

Have you evaluated the model quality? I've always suspected that instruct-tuning is much less data-intensive than most people think.

[tloen]

Could also be worth trying to disable the int8 quantization or increase the matrix rank. Will check tomorrow.

[kesar]

same here.

only changed:
BATCH_SIZE = 256
MICRO_BATCH_SIZE = 5

[tloen]

Fwiw I've been able to eke out some small gains setting LORA_R to 8 instead of 4. Otherwise, seems (until proven otherwise) like both learning rates are perfectly fine.