New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WARNING:root:NaN or Inf found in input tensor. #34
Comments
Hello, I've been watching this job recently. Do you use your own data set? What is your error message? |
@LoveIsAGame Hi, I used my own dataset with tow classes labeled in voc format(xml) and the training encountered the issue I posted. |
can you post the log / err message? |
@jason718 sorry for posting the long error log, it said: 2021-01-13 02:05:49,856 wetectron.utils.miscellaneous INFO: Saving labels mapping into output/labels.json |
I prepared corresponding selective search bboxes as the |
@liaorongfan Can I add your wechat or QQ? It's more convenient to communicate in this way. |
I don't know how to prepare my own data. My data is supervised data. How can I generate his pkl file? |
@LoveIsAGame I guess, we may fully discuss the issues we encountered here. |
@LoveIsAGame Hi , for selective search proposals this repo would be helpful. |
Maybe some items of your dataset/proposal are broken? Have you checked? You can also make the batchsize=1 and print out loss at every iteration to see when the loss become Nan. |
@jason718 Hi, thanks very much for the instructions. The reason caused to that problem may be too few proposals for each images, average in 500/image, and then I select more proposals and it seems OK now. Very excellent work, thanks for sharing. |
I had the same issue, but I fixed it by updating the learning rate. |
thanks,me too |
@JanuaryThomas @Zhangmaopeng88 How did you solve it? Changed the learning rate small? |
Do you increase or decrease the learning rate to solve this problem? |
If anyone suffers from the issue and if you double checked your data, first:
It is a general solution that worked for my 3090 single GPU, hope it helps |
Training form scratch with "V_16_voc07.yaml" with batch size of 1 on one GForce 1080Ti GPU, after 6000 iters, the logger gave this warming, I have no idea where got things wrong. Could you help me with some clue
The text was updated successfully, but these errors were encountered: