-
Notifications
You must be signed in to change notification settings - Fork 8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
yolov4 Training questions nan nan nan nan nan #5395
Comments
What cfg-file do you use?
If you use 4x GPUs, then use 4/ less https://github.com/AlexeyAB/darknet#how-to-train-with-multi-gpu |
still got nan using learning_rate=0.00025@AlexeyAB |
Show screenshot. |
I have a similar issue when I train in a p2.xlarge with a modified COCO dataset |
Did you follow? https://github.com/AlexeyAB/darknet#how-to-train-with-multi-gpu |
I trained 1000 epochs on my pc and pushed everything on an ec2 instance. I think p2.xlarge is not a multi GPU instance anymore. |
|
After having configured batch=32 and subdivisions=16, the training start but after a few iterations, NAN appears xD. I also used a p3.2xlarge instance; same issue. |
|
yolov4.conv.137 |
I just fixed Change And train: |
It works now, thank you @AlexeyAB ! |
@AlexeyAB |
put stopbackward=800 before some layer meaning that all the layers' weights before current layer will not be updated only for the first 800 iterations. |
@AlexeyAB |
@beizhengren Yes! |
@AlexeyAB Thanks! |
@AlexeyAB I followed it .all parameters in yolov4.cfg by default, except for width=512 height=512 Training command: but NAN happened at 10000 iteration.I use 32G GPU What's wrong with it? |
Show screenshot. And attach cfg-file. |
chart.png,loss is always Greater than 10 .So it is a line cfg-file: |
|
|
@Li505358678 I fixed yolov4.cfg for stable training without Nan, it will have almost the same AP: c0d6b81 change |
I will try.What's mean about have almost the same AP?the AP will not decrease and bad? |
Increase instead of decrease |
Accuracy will be the same. |
Thanks!I will try it. |
I have another question. I want to get a model just like yolov4(yolov4.conv.137),Will this training lead to a different model from yolov4? if I want is a model close to yolov4, including the configuration of the cfg file. What should I do? |
Just if you want to reproduce our results from the paper.
I don't understand. Do you want to check our results from paper, or do you want to train your different model on MSCOCO, or do you want to train your own model for your custom objects, ...? |
I want to reproduce your results from the paper.I want to get the same model as yolov4.conv.137. Now the question is, if I follow the https://github.com/AlexeyAB/darknet/wiki/Train-Detector-on-MS-COCO-(trainvalno5k-2014)-dataset. Can I get the model close to yolov4.conv.137? The reason why I have this problem is that the height and width in the cfg-file must be set to 512 during training, but if I directly use yolov4.conv.137 for custom object training or evaluating MSCOCO, I can use any width and height that are multiple of 32. If I train according to the above link(just train on MSCOCO), can the resulting model also be used to evaluate by using any width and height that are multiple of 32 |
Yes.
To reproduce paper results:
|
for the new code ?If I want to train with multi-GPU on MS COCO.Do I need to change the learning_rate and burn_in of the yolov4.cfg |
Training has been more than 10000 times, why the loss is still hundreds |
Show chart.png
I would recommend to use for 4xGPUs |
the chart.png just can see a line on the top command:./darknet detector train cfg/coco.data cfg/yolov4.cfg csdarknet53-omega.conv.105 just train on MSCOCO follow https://github.com/AlexeyAB/darknet/wiki/Train-Detector-on-MS-COCO-(trainvalno5k-2014)-dataset If I just have 1GPU. What should I do? |
Just follow: https://github.com/AlexeyAB/darknet/wiki/Train-Detector-on-MS-COCO-(trainvalno5k-2014)-dataset |
Yes.I follow it . and after 10000 times. the loss was also high that over 100. is this normal? |
Loss should be ~20. But show chart.png |
|
It's normal at the beginning of training. After a while of training, the loss value suddenly increases, and then it's all Nan.
![image](https://user-images.githubusercontent.com/45789648/80555044-9fdeca80-8a01-11ea-98bf-b163825f4ac4.png)
![image](https://user-images.githubusercontent.com/45789648/80555062-b8e77b80-8a01-11ea-993d-7025882604a6.png)
According to the current check, there is no problem with the data set, and the learning rate is set to 0.001,who can help me?thanks
The text was updated successfully, but these errors were encountered: