[Bug]

[zzx0226]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• The bug has not been fixed in the latest version (0.x) or latest version (1.x).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

1.x branch https://github.com/open-mmlab/mmocr/tree/dev-1.x

Environment

addict 2.4.0
brotlipy 0.7.0
certifi 2022.6.15
cffi 1.15.0
charset-normalizer 2.0.4
click 8.1.3
colorama 0.4.6
commonmark 0.9.1
cryptography 37.0.1
cycler 0.11.0
fonttools 4.38.0
idna 3.3
imageio 2.22.4
imgaug 0.4.0
importlib-metadata 5.1.0
kiwisolver 1.4.4
lanms-neo 1.0.2
lmdb 1.3.0
Markdown 3.4.1
matplotlib 3.5.3
mkl-fft 1.3.1
mkl-random 1.2.2
mkl-service 2.4.0
mmcv 2.0.0rc3
mmcv-full 1.7.0
mmdet 3.0.0rc4
mmengine 0.3.2
mmocr 1.0.0rc3
model-index 0.1.11
networkx 2.6.3
numpy 1.21.5
opencv-python 4.6.0.66
openmim 0.3.3
ordered-set 4.1.0
packaging 21.3
pandas 1.3.5
Pillow 9.2.0
pip 22.1.2
pyclipper 1.3.0.post4
pycocotools 2.0.6
pycparser 2.21
Pygments 2.13.0
pyOpenSSL 22.0.0
pyparsing 3.0.9
PySocks 1.7.1
python-dateutil 2.8.2
pytz 2022.6
PyWavelets 1.3.0
PyYAML 6.0
rapidfuzz 2.13.2
requests 2.28.1
rich 12.6.0
scikit-image 0.19.3
scipy 1.7.3
setuptools 61.2.0
Shapely 1.8.5.post1
six 1.16.0
tabulate 0.9.0
termcolor 2.1.1
terminaltables 3.1.10
tifffile 2021.11.2
torch 1.12.0
torchaudio 0.12.0
torchvision 0.13.0
typing_extensions 4.1.1
urllib3 1.26.9
wheel 0.37.1
yapf 0.32.0
zipp 3.11.0

Reproduces the problem - code sample

python tools/train.py configs/textdet/fcenet/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500.py --resume

Reproduces the problem - command or script

python tools/train.py configs/textdet/fcenet/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500.py --resume

Reproduces the problem - error message

Traceback (most recent call last):
File "tools/train.py", line 117, in
main()
File "tools/train.py", line 113, in main
runner.train()
File "/home/ubuntu/anaconda3/envs/Detr/lib/python3.7/site-packages/mmengine/runner/runner.py", line 1684, in train
model = self.train_loop.run() # type: ignore
File "/home/ubuntu/anaconda3/envs/Detr/lib/python3.7/site-packages/mmengine/runner/loops.py", line 90, in run
self.run_epoch()
File "/home/ubuntu/anaconda3/envs/Detr/lib/python3.7/site-packages/mmengine/runner/loops.py", line 106, in run_epoch
self.run_iter(idx, data_batch)
File "/home/ubuntu/anaconda3/envs/Detr/lib/python3.7/site-packages/mmengine/runner/loops.py", line 123, in run_iter
data_batch, optim_wrapper=self.runner.optim_wrapper)
File "/home/ubuntu/anaconda3/envs/Detr/lib/python3.7/site-packages/mmengine/model/base_model/base_model.py", line 116, in train_step
optim_wrapper.update_params(parsed_losses)
File "/home/ubuntu/anaconda3/envs/Detr/lib/python3.7/site-packages/mmengine/optim/optimizer/optimizer_wrapper.py", line 176, in update_params
self.step()
File "/home/ubuntu/anaconda3/envs/Detr/lib/python3.7/site-packages/mmengine/optim/scheduler/param_scheduler.py", line 115, in wrapper
return wrapped(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/Detr/lib/python3.7/site-packages/mmengine/optim/optimizer/optimizer_wrapper.py", line 228, in step
self.optimizer.step(**kwargs)
File "/home/ubuntu/anaconda3/envs/Detr/lib/python3.7/site-packages/torch/optim/optimizer.py", line 109, in wrapper
return func(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/Detr/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/Detr/lib/python3.7/site-packages/torch/optim/adamw.py", line 175, in step
capturable=group['capturable'])
File "/home/ubuntu/anaconda3/envs/Detr/lib/python3.7/site-packages/torch/optim/adamw.py", line 231, in adamw
capturable=capturable)
File "/home/ubuntu/anaconda3/envs/Detr/lib/python3.7/site-packages/torch/optim/adamw.py", line 259, in _single_tensor_adamw
assert not step_t.is_cuda, "If capturable=False, state_steps should not be CUDA tensors."
AssertionError: If capturable=False, state_steps should not be CUDA tensors.

Additional information

mmocr 1.x resume功能无法使用，希望早日修复。
感谢！

[Harold-lkk]

Thanks for your report! It's actually PyTorch's problem. Refer to pytorch/pytorch#80809

[zzx0226]

Thanks for your report! It's actually PyTorch's problem. Refer to pytorch/pytorch#80809

Thanks for your relpy !
I downgrade pytorch to 1.11.0 then reinstall mmcv-full mmdeteion and mmocr, the problem has been fixed.