Reproducing loss

[Luvata]

Hi @rmokady, what a clever approach!

I'm trying this approach on my custom dataset and manage to get it start training. I'm figuring out way to add evaluate code to better manage the training, but in the mean time, I wonder what is your loss score when stop training the model on each mode: Train only prefix and Train both prefix and GPT?

[rmokady]

Hi @Luvata , for COCO where we train both prefix and GPT-2 the loss got to 1.47
Hope this is helpful

[Luvata]

Thank you @rmokady, that was very helpful