继续预训练如何加载模型？

[fade-color]

我在pretrain_glm.py继续预训练加载下载下来的glm-large-chinese/mp_rank_00_model_states.pt时报错：

WARNING: could not find the metadata file /root/Data/zz/GitHub/GLM/blocklm-large-chinese/latest_checkpointed_iteration.txt 
Try to directly load the checkpoint from the directory
Traceback (most recent call last):
  File "pretrain_glm.py", line 663, in <module>
    main()
  File "pretrain_glm.py", line 580, in main
    args.iteration = load_checkpoint(model, optimizer, lr_scheduler, args)
  File "/root/Data/zz/GitHub/GLM/utils.py", line 337, in load_checkpoint
    checkpoint_name, sd = model.load_checkpoint(load_dir, tag,
  File "/root/anaconda3/envs/deepspeed/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2513, in load_checkpoint
    load_path, client_states = self._load_checkpoint(load_dir,
  File "/root/anaconda3/envs/deepspeed/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2671, in _load_checkpoint
    client_state['optimizer'] = optim_checkpoint['optimizer']
KeyError: 'optimizer'

提供的模型文件要怎么才能正确加载呢？

[fade-color]

我尝试设置--no-load-optim，但是发现没有用

[duzx16]

DeepSpeed's load_checkpoint function will always load the optimizer state even if load_optimizer_states=False.
You can pull the latest commit and set --no-deepspeed-load.

[fade-color]

谢谢，可以了

[zhangzai666]

谢谢，可以了

请问您的预训练数据格式可以参考一下吗