Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

when i training my data,i get a error(ValueError: cannot convert float NaN to integer).i dont konw why? #13

Closed
gavinzhang1995 opened this issue Nov 14, 2017 · 13 comments

Comments

@gavinzhang1995
Copy link

I1114 21:43:12.214639 28872 sgd_solver.cpp:106] Iteration 0, lr = 0.0002
/home/gavin/icpr/FPN-caffe/tools/../lib/rpn/proposal_target_layer.py:225: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
fg_inds = npr.choice(fg_inds, size=fg_rois_per_this_image, replace=False)
/home/gavin/icpr/FPN-caffe/tools/../lib/rpn/proposal_target_layer.py:236: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
bg_inds = npr.choice(bg_inds, size=bg_rois_per_this_image, replace=False)
/home/gavin/icpr/FPN-caffe/tools/../lib/rpn/proposal_target_layer.py:243: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
labels[fg_rois_per_this_image:] = 0
/home/gavin/icpr/FPN-caffe/tools/../lib/rpn/anchor_target_layer.py:104: RuntimeWarning: invalid value encountered in sqrt
k = np.floor(k0 + np.log2(np.sqrt(areas)*1.0 / 64))
/home/gavin/icpr/FPN-caffe/tools/../lib/rpn/anchor_target_layer.py:106: RuntimeWarning: invalid value encountered in less
inds = np.where(k < min_k)[0]
/home/gavin/icpr/FPN-caffe/tools/../lib/rpn/anchor_target_layer.py:108: RuntimeWarning: invalid value encountered in greater
inds = np.where(k > max_k)[0]
Traceback (most recent call last):
File "./tools/train_net.py", line 112, in
max_iters=args.max_iters)
File "/home/gavin/icpr/FPN-caffe/tools/../lib/fast_rcnn/train.py", line 160, in train_net
model_paths = sw.train_model(max_iters)
File "/home/gavin/icpr/FPN-caffe/tools/../lib/fast_rcnn/train.py", line 101, in train_model
self.solver.step(1)
File "/home/gavin/icpr/FPN-caffe/tools/../lib/rpn/anchor_target_layer.py", line 116, in forward
if int(item)==p:
ValueError: cannot convert float NaN to integer
Command exited with non-zero status 1

@busyboxs
Copy link

busyboxs commented Nov 16, 2017

@gavinzhang1995
as to VisibleDeprecationWarning, maybe you should change code as follow:
fg_inds = npr.choice(fg_inds, size=int(fg_rois_per_this_image), replace=False)
bg_inds = npr.choice(bg_inds, size=int(bg_rois_per_this_image), replace=False)
labels[int(fg_rois_per_this_image):] = 0

@cs-heibao
Copy link

@gavinzhang1995
Did you solve the error?

@cs-heibao
Copy link

@busyboxs @gavinzhang1995
I found that the error caused by this line of code “ k = np.floor(k0 + np.log2(np.sqrt(areas)1.0 / 64))",and the mainly problem is that the value of areas sometimes maybe negtive, it's very strange!
The error occurred when Iteration 200, and I print the areas:
I1117 21:20:22.956712 20135 solver.cpp:229] Iteration 200, loss = 0.981587
I1117 21:20:22.956739 20135 solver.cpp:245] Train net output #0: loss_bbox = 0.0910424 (
1 = 0.0910424 loss)
I1117 21:20:22.956758 20135 solver.cpp:245] Train net output #1: loss_cls = 0.268306 (* 1 = 0.268306 loss)
I1117 21:20:22.956763 20135 solver.cpp:245] Train net output #2: rpn_cls_loss/p2 = 0.0182713 (* 1 = 0.0182713 loss)
I1117 21:20:22.956768 20135 solver.cpp:245] Train net output #3: rpn_cls_loss/p3 = 0.107832 (* 1 = 0.107832 loss)
I1117 21:20:22.956784 20135 solver.cpp:245] Train net output #4: rpn_cls_loss/p4 = 0.0701596 (* 1 = 0.0701596 loss)
I1117 21:20:22.956787 20135 solver.cpp:245] Train net output #5: rpn_cls_loss/p5 = 0.128425 (* 1 = 0.128425 loss)
I1117 21:20:22.956806 20135 solver.cpp:245] Train net output #6: rpn_loss_bbox/p2 = 0 (* 1 = 0 loss)
I1117 21:20:22.956810 20135 solver.cpp:245] Train net output #7: rpn_loss_bbox/p3 = 0.00275289 (* 1 = 0.00275289 loss)
I1117 21:20:22.956815 20135 solver.cpp:245] Train net output #8: rpn_loss_bbox/p4 = 0.00327946 (* 1 = 0.00327946 loss)
I1117 21:20:22.956818 20135 solver.cpp:245] Train net output #9: rpn_loss_bbox/p5 = 0 (* 1 = 0 loss)
I1117 21:20:22.956823 20135 sgd_solver.cpp:106] Iteration 200, lr = 0.0002
[ 98314.6796875]
[ 4973.14453125 1771.54541016 1843.26171875 1879.8828125 4442.74902344]
[ 4973.14453125 1771.54541016 1843.26171875 1879.8828125 4442.74902344]
[-6696563.]
and there is a negative number, I don't know why

@busyboxs
Copy link

@JunJieAI
I trained network with pascal voc dataset, and there is no problem. Which dataset did you use?

@cs-heibao
Copy link

@busyboxs
You used the public pascal voc data or your own data? I used my own data

@RoseLii
Copy link

RoseLii commented Nov 27, 2017

Did you solve the error? @gavinzhang1995 I also had this problem.

@busyboxs
Copy link

@RoseLii
what about change TRAIN.RPN_MIN_SIZE = 16 in "lib/fast_rcnn/config.py" or set it in "FP_Net_end2end.yml", you can have a try.

@RoseLii
Copy link

RoseLii commented Nov 27, 2017

thank you.i will try the way. @busyboxs

@RoseLii
Copy link

RoseLii commented Nov 27, 2017

@busyboxs excuse me.我在训练VOC2007时,整个训练过程是没有问题的,但是我在训练自己的数据集时出现了这个问题。由于我的数据集中有些图片是没有标签的,所以我将那部分的标签设为0 0 0 0,然后生成xml文件,然后就报错了,不知道怎么改 @JunJieAI 你的问题解决了吗?帮帮我

@gavinzhang1995
Copy link
Author

so sorry,i dont solve this error.i also use my own data. @JunJieAI @busyboxs @RoseLii @unsky

@unsky
Copy link
Owner

unsky commented Dec 19, 2017

i have update codes

@unsky unsky closed this as completed Dec 20, 2017
@sujinpeng
Copy link

Did you solve the error? @gavinzhang1995 I meet the problem,thank you

@sujinpeng
Copy link

could you tell me what is 64?why not 224?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants