Finetuning the Galois' model

[DenisAraujo68]

Hi, @iedmrc
I'm finetuning the Galois' model with the gpt-2-simple command aiming at featuring it with our team programming standards. (Well, actually, we hope so!)
I'm running the finetune with "steps=-1" (what's, endless run).
I'd like to hear from you when should I stop the process.
This is the last 4 lines of the current history of the process:

[310 | 23899.61] loss=0.09 avg=0.36
[320 | 24645.14] loss=0.06 avg=0.35
[330 | 25398.12] loss=0.09 avg=0.34
[340 | 26155.55] loss=0.05 avg=0.33

Best regard!

[iedmrc]

Hi,
The loss of the network is yet another evaluation metric as you know. And we do not have much more or good metrics in auto-regressive models for now (as I know). Having such low losses may mean the network is overfitting. The GPT2 is so much powerful and if you don't have enough data to feed, it'll quickly overfit to your data and it just predicts the words in your dataset only.

Another metric is to calculate perplexity but it's not included in gpt-2-simple yet. You may want to calculate it yourself: https://en.wikipedia.org/wiki/Perplexity