Train matching pretrained checkpoint for new dataset

[edbltn]

Hi all,

Some classmates and I are trying to train a fusion text generation model using the fairseq code, but on a new dataset that we have scraped and cleaned. We are struggling with how to train a usable pretrained checkpoint.

When I used the checkpoint (pretrained_checkpoint.pt) provided in the README, I got the following error:

RuntimeError: Error(s) in loading state_dict for FConvModelSelfAtt:
        size mismatch for encoder.encoder.embed_tokens.weight: copying a param of torch.Size([3257, 256]) from checkpoint, where the shape is torch.Size([19025, 256]) in current model.
        size mismatch for decoder.embed_tokens.weight: copying a param of torch.Size([51411, 256]) from checkpoint,
 where the shape is torch.Size([104960, 256]) in current model.
        size mismatch for decoder.fc3.weight: copying a param of torch.Size([51411, 256]) from checkpoint, where th
e shape is torch.Size([104960, 256]) in current model.
        size mismatch for decoder.fc3.bias: copying a param of torch.Size([51411]) from checkpoint, where the shape
 is torch.Size([104960]) in current model.

Here is the line of code that I ran, where "dummy_source" and "dummy_target" are our input and output languages.

    $ python train.py data-bin/dummy -a fconv_self_att_wp --lr 0.25 \
        --clip-norm 0.1 --max-tokens 1500 --lr-scheduler reduce_lr_on_plateau \
        --decoder-attention True --encoder-attention False --criterion \
        label_smoothed_cross_entropy --weight-decay .0000001 --label-smoothing 0 \
        --source-lang dummy_source --target-lang dummy_target --gated-attention True \
        --self-attention True --project-input True --pretrained True \
        --pretrained-checkpoint data-bin/models/pretrained_checkpoint.pt

But when I attempted to use a checkpoint created by training on our new dataset (checkpoint_best.pt) I got:

Exception: Cannot load model parameters from checkpoint, please ensure that the architectures match

Here is the line of code I ran for that:

    $ python train.py data-bin/dummy -a fconv_self_att_wp --lr 0.25 \
        --clip-norm 0.1 --max-tokens 1500 --lr-scheduler reduce_lr_on_plateau \
        --decoder-attention True --encoder-attention False --criterion \
        label_smoothed_cross_entropy --weight-decay .0000001 --label-smoothing 0 \
        --source-lang dummy_source --target-lang dummy_target --gated-attention True \
        --self-attention True --project-input True --pretrained True \
        --pretrained-checkpoint checkpoints/checkpoint_best.pt

How can we go about training a usable pretrained checkpoint on a new dataset?

[jerinphilip]

You have to reuse the dictionary from the old dataset to build the new binary dataset. You'll thus end with the same number of vocabulary classes, making the models match. The flags are in preprocess.py

[edbltn]

Jerin, thanks for the reply. That would enable us to use the existing pretrained_checkpoint.pt file, which I take it was trained on the writingPrompts dataset. What if we want to train our own pretrained_checkpoint.pt?

[jerinphilip]

If you don't supply a dictionary file, it'll build its own and build a binary dataset on top of it. This gets you training from scratch.

…

On Tue, 20 Nov 2018, 08:58 edbltn ***@***.*** wrote: Jerin, thanks for the reply. That would enable us to use the existing pretrained_checkpoint.pt file, which I take it was trained on the writingPrompts dataset. What if we want to train our own pretrained_checkpoint.pt? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#367 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAsY_PvCNRZX8zd6SWSwSYMNKdnuKCcBks5uw3bWgaJpZM4YfNma> .

[edbltn]

If we do that and get a model trained from scratch, how can we then use that as a pretrained checkpoint for training a fusion model?

[huihuifan]

Hi, sorry I missed this. The pretrained model is simply another seq2seq model. So you would train a seq2seq model on your data, and save this as the pretrained model. Then you train your second seq2seq model, passing this pretrained checkpoint. You can see some example commands here: https://github.com/fairinternal/fairseq-py/tree/master/examples/stories

Hope that helps! Make sure that the pretrained model and your training model have the same vocabulary size, or they won't be able to fuse together.

[onyalcin]

Hi, I have the same problem now while trying to train the model on our own data. I followed the example commands here: https://github.com/pytorch/fairseq/tree/master/examples/stories

First, trained the model on our own data without pretrained checkpoint:
$ python train.py data-bin/our_data -a fconv_self_att_wp --lr 0.25 --clip-norm 0.1 --max-tokens 1500 --lr-scheduler reduce_lr_on_plateau --decoder-attention True --encoder-attention False --criterion label_smoothed_cross_entropy --weight-decay .0000001 --label-smoothing 0 --source-lang wp_source --target-lang wp_target --gated-attention True --self-attention True --project-input True --pretrained False

And then followed up to train the fusion model:
$ python train.py data-bin/our_data -a fconv_self_att_wp --lr 0.25 --clip-norm 0.1 --max-tokens 1500 --lr-scheduler reduce_lr_on_plateau --decoder-attention True --encoder-attention False --criterion label_smoothed_cross_entropy --weight-decay .0000001 --label-smoothing 0 --source-lang wp_source --target-lang wp_target --gated-attention True --self-attention True --project-input True --pretrained True --pretrained-checkpoint ./checkpoints/checkpoint_best.pt

Which gave me the following error:
Exception: Cannot load model parameters from checkpoint, please ensure that the architectures match

Is there something specific I am missing while training the first model?

[huihuifan]

Are you sure the first model is being saved in: ./checkpoints/checkpoint_best.pt ? The error looks like this "./checkpoints/checkpoint_best.pt" model isn't an fconv_self_att_wp model, so it's having trouble loading. If you want, you can examine the state dict file of "./checkpoints/checkpoint_best.pt" and see what's in it.

[onyalcin]

Yes, I am sure, it gives this output while training:

`| loading pretrained model

FConvModelSelfAtt(
(encoder): CompositeEncoder(

(encoder): FConvEncoder(
  (embed_tokens): Embedding(108, 256, padding_idx=1)
  (embed_positions): LearnedPositionalEmbedding(1024, 256, padding_idx=1)
  (fc1): Linear(in_features=256, out_features=128, bias=True)
  (projections): ModuleList(
    (0): None
    (1): None
    (2): Linear(in_features=128, out_features=512, bias=True)`  ...

(not copying all the output but I can share it if you think it is important)

[Arafat4341]

Hello! I trained the model with JSUT data. Training was completed upto checkpoint_step000100000.pth. And while synthesizing I get:

size mismatch for seq2seq.encoder.embed_tokens.weight: copying a param with shape torch.Size([65535, 256]) from checkpoint, the shape in current model is torch.Size([149, 256]).

Can you tell why am I getting this issue?

[jerinphilip]

The shape is `vocab_size × embedding_size`. So you're supplying a different dictionary at inference time? 65535 is vocab size what you have trained with. 149 of the dictionary supplied for inference seems off unless you're doing some character level task - if training is correct this should also come to be 65535 with the same dictionaries being used for inference. You can find command line arguments for supplying dicts somewhere in the docs.

…

-- Jerin Philip

________________________________ From: Md Yeasin Arafath <notifications@github.com> Sent: Wednesday, January 1, 2020 10:00:27 AM To: pytorch/fairseq <fairseq@noreply.github.com> Cc: Jerin Philip <jerin.philip@research.iiit.ac.in>; Comment <comment@noreply.github.com> Subject: Re: [pytorch/fairseq] Train matching pretrained checkpoint for new dataset (#367) Hello! I trained the model with JSUT data. Training was completed upto checkpoint_step000100000.pth. And while synthesizing I get: size mismatch for seq2seq.encoder.embed_tokens.weight: copying a param with shape torch.Size([65535, 256]) from checkpoint, the shape in current model is torch.Size([149, 256]). Can you tell why am I getting this issue? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#367>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAFRR7CFLKA4ZKTOZHH2N7TQ3QL6HANCNFSM4GD43GNA>.

[Arafat4341]

@jerinphilip Thanks a lot for responding!
Yes I supplied a different text file to synthesize. Training was completed upto checkpoint_step000100000.pth. As I use colab, I think the runtime expired. Because It took a long time although I used GPU.
Now can't I use the training upto this step?
Can't I use my model to synthesize custom texts?
And how come dicts help in this case?

[Arafat4341]

@jerinphilip does that mean my training hasn't been accurate?