-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error occured during training BasicVSR #273
Comments
Did you use the provided configuration? This error occur because some trainable parameters do not receive gradient. |
Yes, I use configuration in "configs/restorers/basicvsr/basicvsr_reds4.py". if I don't add |
I just tried and did not encounter the error. Could you provide me with more details? What command do you use for training? And did you modify any codes in the model or architecture? |
Environment: command for training: ./tools/dist_train.sh ./configs/restorers/basicvsr/basicvsr_reds4.py 2 I double check configuation file and make sure that I use official configuation. And here is my training log.
No, I just remove |
Could you revert to the original code (don't add or remove |
I tried original code with |
I just tried to run Can you please try to print out the value of |
It is |
It is |
Yes...I'm also very confused about that. |
It seems that in the repository "MMDetection", someone encountered similiar issue |
That's strange, I will look into that. |
Thanks for your your detailed reply. |
@NK-CS-ZZL Hi, would you be able to test whether the problem exists in PyTorch |
I tried it and code works well. After 5000 iterations, Spynet updates as expected. |
Then it's probably because of the difference in the PyTorch version. I will further look into the problem. Thanks for your report. |
Fixed in #290. Thank you for your report. Please reopen there are still any issues. |
During training, error as follows occured. I run it with official config in "configs/restorers/basicvsr/basicvsr_reds4.py".
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument "find_unused_parameters=True" to "torch.nn.parallel.DistributedDataParallel"; (2) making sure all "forward" function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's "forward" function. Please include the loss function and the structure of the return value of "forward" of your module when reporting this issue (e.g. list, dict, iterable).
Though I solved it by adding
find_unused_parameters=True
in line 128, "apis/train.py" and removingself.generator.find_unused_parameters = False
in line 84, "models/restorers/basicvsr.py", I still don't know why that error happened.In "models/restorers/basicvsr.py", author declares a variable "self.generator.find_unused_parameters = False", which seems that shouldn't have influence on program, for it's never used as a parameter in program. However, it indeed causes the error mentioned above.
The text was updated successfully, but these errors were encountered: