how to pretrain a ctrl model from scratch ?

[xurongqiang]

We wanna pretrain a ctrl model from scratch, could you provide some implementation details?
What is the format of the training sample and can the training process be finished with script training.py ?

[keskarnitish]

The training_utils folder should be a great starting point here.
Collect your data (Wikipedia/Reddit/News/etc.), convert to TFRecords with the appropriate control codes, transfer the data to GCS if using TPUs, and then train. We used Adagrad with a linear warmup and no learning rate decay. With TPUs, you can spawn a cloud TPU pod and the Estimator will take care of all data parallelism.

[xurongqiang]

The training_utils folder should be a great starting point here.
Collect your data (Wikipedia/Reddit/News/etc.), convert to TFRecords with the appropriate control codes, transfer the data to GCS if using TPUs, and then train. We used Adagrad with a linear warmup and no learning rate decay. With TPUs, you can spawn a cloud TPU pod and the Estimator will take care of all data parallelism.

How are these two files（codes && control_codes.txt） generated?

[keskarnitish]

codes has nothing to do with the control codes; it is the BPE codes you get from fastBPE (see https://github.com/glample/fastBPE#learn-codes)

For control_codes.txt, you can first collect your data and decide the list of control codes you want (this is the first column). Then, TFRecord each file with its corresponding control code map and then figure out the percentage of data later (if this is relevant to you; for training it isn't).

[leejason]

Is it feasible to train a CTRL model from scratch on Colab with free TPU? If not, how many TPUs would be required for how much money?

[keskarnitish]

On a large amount of data, I don't think that would work. We trained on 256 cores of the Cloud TPU v3 Pod. You should be able to train on slightly smaller slices with a commensurate increase in training time as well. Regd. pricing, I think the best resource would be Google's official sheet.