Pretrain from scratch

[agemagician]

Hi,

I am trying to pretrain from scratch and I am using the fine-tune colab example as a base for my code.
Everything runs fine except for the train part.

I have changed the train part to :

TRAIN_STEPS = 25000 #@param {type: "integer"}

model.train(
    mixture_or_task_name="ss3",
    steps = TRAIN_STEPS
)

It shows me the following error:
Required bindings for make_layer_stack not provided in config: ['layers']

I would assume this is a problem because I didn't define the model configuration.

Could you please let us know how you can adjust the fine-tune colab example to pretrain example from scratch ?

[craffel]

When one calls model.finetune as is done in the colab, the operative gin config file which resides in the pretrained model directly is automatically loaded.
https://github.com/google-research/text-to-text-transfer-transformer/blob/master/t5/models/mtf_model.py#L308
Parsing this config file provides values for things like make_layer_stack. When using model.train, there is no pre-trained model directory so there is no operative config to look at. One option would be to decide on a model you want to use and load the operative config for that file, e.g. for the Base model

    import gin
    with gin.unlock_config():
      gin.parse_config_file("gs://t5-data/pretrained_models/base/operative_config.gin")

Note that you still may need to provide or modify other gin parameters.