Are the training loss and validation loss recorded?

[shatealaboxiaowang]

Hi, Dear:

Thank you very much for your code. I am reproducing your training process. I wonder what your training loss and validation loss are during the training process, and I want to align them with your training process on dataset Magicoder-OSS-Instruct-75K and datasets--ise-uiuc--Magicoder-Evol-Instruct-110K

Thx

[shatealaboxiaowang]

Thank you for your open source, I am replicating your fine-tuning process according to the code on github. Do the results of train loss=0.16 and eval_loss=0.21 I trained on the 75k dataset match yours? I will continue training on the 110k dataset.
I trained for 4 epochs and indeed started overfitting after the second epoch.

[UniverseFly]

Magicoder-S-CL.json
Magicoder-CL.json
Magicoder-S-DS.json
Magicoder-DS.json

Hi, here are the trainer states. Hope they can help!

[shatealaboxiaowang]

Magicoder-S-CL.json Magicoder-CL.json Magicoder-S-DS.json Magicoder-DS.json

Hi, here are the trainer states. Hope they can help!

Thank you very much, my training process is basically the same as your loss metric and the test results on humaneval dataset are basically consistent

But I have a question. Why is the model fully fine-tuned by instruction, but why is the capacity infilinig increased?
I look forward to hearing from you.

[UniverseFly]

Good to hear you can reproduce it. Yeah we did observe that the infilling capability was at least not decreasing. We believe this is because the model learned some general alignment during the instruction tuning, and infilling is a kind of alignment based on the surrounding context. Further study of this phenomenon would be interesting.