-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Predicted boxes or scores contain Inf/NaN. Training has diverged #4
Comments
It may be problem of detectron version, I will try detectron0.3 next |
problem solved |
I am facing the same problem as you. Can you tell me how you solved it? |
using detectron0.3,the default version is 0.5 |
How to install detectron0.3 when i use the server with cuda11.1 |
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When trying to run the training code, I encountered the following error:
![image](https://user-images.githubusercontent.com/43271691/137419767-ab21d493-9372-46e0-8975-051b238d371e.png)
In the previous process, my steps were:
Refer to the readme, install torch, detectron and other environment dependencies based on conda
Data preparation: vocdata in readme is used
![image](https://user-images.githubusercontent.com/43271691/137421025-80bbc0fe-3038-4e19-8b41-4cb64314ded4.png)
Modify operating errors:
![image](https://user-images.githubusercontent.com/43271691/137421226-4b48b4cc-2776-4673-8239-709f441ed6f9.png)
![image](https://user-images.githubusercontent.com/43271691/137421392-9e6adafa-cfb5-4628-a035-73ce73300f8b.png)
1、in the run_*.sh ,change into
2、as the picture shows(Refer to:https://giters.com/fanq15/FewX/issues/40?amp=1)
Intercept part of the training code and try to start training
![image](https://user-images.githubusercontent.com/43271691/137421608-6620e40b-33eb-4a15-9d55-a26e7a03041b.png)
then i meet the error:Predicted boxes or scores contain Inf/NaN. Training has diverged。I got the same error when I tried on two different servers (cuda10.2, cuda11.1 both torch1.8)
I tried to modify the learning rate-related settings(to different value , vary from 0.00005 to 0.95) in the configuration file, but no effect
![image](https://user-images.githubusercontent.com/43271691/137422053-ffb3d14a-f05c-47dc-a8e7-ac82aada1fa3.png)
The text was updated successfully, but these errors were encountered: