Pretrain and generate

[king-menin]

if i run with
python -m torch.distributed.launch --nproc_per_node 16 pretrain_gpt2.py --model_parallel_size==16
and after run generate:
python generate_samples.py
i have error while initialization: size mismatch for transformer.layers.15.attention.dense.weight: copying a param with shape torch.Size([1024, 64]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
can i load model on one GPU and train distributed on 16 GPU with --model_parallel_size==16?
thank you!

[github-actions]

Marking as stale. No activity in 60 days. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

Marking as stale. No activity in 60 days.