how to train it

[jackylee1]

how to train it

[miweru]

that code is not released and maybe wont be released

[branc116]

Can you create a model that accepts other languages (e.g. German, Russian)?

[WuTheFWasThat]

@jackylee1 Sorry, as others mentioned, we're not releasing training code for now, as this repository is currently for exploring the existing model. You may find the model.py code in this repo useful, and there are plenty of other projects that train transformer models out there!

@branc116 we're not taking requests to train any models as we have limited bandwidth (especially if they require procuring a dataset differently!)

[marcpre]

@openai
Impressive results!

@WuTheFWasThat
Any suggestions for other transformer models, that can be trained and played around?

[guotong1988]

Same question. Thank you.

[WuTheFWasThat]

RE other pretrained models: definitely check out Google's BERT model and CMU/Google's Transformer-XL models

[gwern]

nshepperd has released training code for retraining GPT-2-small: https://github.com/nshepperd/gpt-2/tree/finetuning

It works: I've used it for retraining on anime plot synopses & Project Gutenberg poetry.

[guotong1988]

Thank you @gwern

[kaihuchen]

@nshepperd @gwern Thank you so much for sharing the training code (ref: https://github.com/nshepperd/gpt-2/tree/finetuning) !

I was able to train a model (using Chinese dataset) over the default 117M model without problem. However, when I try to generate samples (either conditional or unconditional), I got "FileNotFoundError: [Errno 2] No such file or directory" for encoder.json, hparams.json and vocab.bpe.

What would be the correct procedure for generating samples for the new model described above?

[gwern]

I am not entirely sure because you said you are using Chinese rather than English text.

For English text, all you need to do is copy over those missing files from the 117M model directory, which will have them (assuming you ran the download script as you must've if you did any retraining of 117M, of course).

Since finetuning doesn't affect the BPE encoding details, it is merely further training the Transformer model itself; so, the model still assumes the exact same encoding as for OA's 117M model, and the encoding is defined by those files.

[PapayasTehSkeletor]

@gwern @nshepperd
Hey Gwen, I'm a little noob in all of this... Do you know how to use finetuning in Colab? That is, if there is a way to use it in Colab...

[gwern]

I don't know of any reason you couldn't do finetuning in Colab (the main restriction I'm aware of is you only get something like 12 GPU-hours? which is more than enough for many finetuning tasks). But I have little familiarity with it or interest in setting up a notebook to do the finetuning. Colab seems like a very restrictive tool compared to running on your own machine.