You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am training with COCO pretrained YOLOv5m model (specifying load_from=xxx in the config) on a custom dataset, after training for a while, the training stopped on exception. I now try with --resume to resume the training. It apparently loads load_from other than saved ckpt under work_dir. I now need to remove load_from from config to resume. So what is the suggested way to do this?
Traceback (most recent call last):
File "tools/train.py", line 106, in <module>
main()
File "tools/train.py", line 102, in main
runner.train()
File "/anaconda3/envs/mm2_cu11/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1653, in train
self.load_or_resume()
File "/anaconda3/envs/mm2_cu11/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1599, in load_or_resume
self.resume(resume_from)
File "/anaconda3/envs/mm2_cu11/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1954, in resume
self.message_hub.load_state_dict(checkpoint['message_hub'])
KeyError: 'message_hub'
/anaconda3/envs/mm2_cu11/lib/python3.8/site-packages/mmengine/runner/runner.py:1948: UserWarning: The dataset metainfo from the resumed checkpoint is different from the current training dataset, please check the correctness of the checkpoint or the training dataset.
If load_from and resume=True are both set, only load_from will take effect according to MMEngine docs. It could be more convenient.
I already sent feedback to mmengine developers. They will discuss it later.
Prerequisite
🐞 Describe the bug
I am training with COCO pretrained YOLOv5m model (specifying
load_from=xxx
in the config) on a custom dataset, after training for a while, the training stopped on exception. I now try with--resume
to resume the training. It apparently loadsload_from
other than saved ckpt underwork_dir
. I now need to removeload_from
from config to resume. So what is the suggested way to do this?Environment
Additional information
No response
The text was updated successfully, but these errors were encountered: