RuntimeWarning: overflow encountered in exp

[jackytu256]

running test.resnet50_test and encounter this issue.

test/../libs/boxes/bbox_transform.py:61: RuntimeWarning: overflow encountered in exp pred_w = np.exp(dw) * widths[:, np.newaxis]

test/../libs/boxes/bbox_transform.py:61: RuntimeWarning: overflow encountered in multiply pred_w = np.exp(dw) * widths[:, np.newaxis]

[CharlesShang]

That's because of some invalid boxes (large output number, e.g., np.exp(200, dtype=np.float32) ).
I'll fix that.

[CharlesShang]

@jackytu256 Hi, did you run this code on the lastest commit? Have you ever encountered these errors?

I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcupti.so.8.0 locally
E tensorflow/core/common_runtime/bfc_allocator.cc:244] tried to allocate 0 bytes
W tensorflow/core/common_runtime/allocator_retry.cc:32] Request to allocate 0 bytes
F tensorflow/core/common_runtime/gpu/gpu_device.cc:104] EigenAllocator for GPU ran out of memory when allocating 0. See error logs for more detailed info.

[jackytu256]

@CharlesShang
I ran the code based on the code of the latest commit , but didnt encounter these errors(just get None value from update_op op_loss,tot_loss =sess.run([update_op, total_loss] ).

[CharlesShang]

I guess it's only because my platform ....
I've spent whole weekends on this :-(
I'll check my environment.
Can you post your platforms? like cuda and cudnn version, nvidia driver version, tf version
Thank you for your feedback.

[jackytu256]

sure.

platform : Ubuntu 14.04.5 LTS
Cuda compilation tools, release 8.0, V8.0.44
nvidia driver version : 367.57
tf version : 1.0.1
GPU : GTX1080

plz let me know if you have any question regrading to this info. Thanks

[maxenceliu]

@jackytu256
exactly same platform as yours, and exactly same problem like yours. Have you fix it?

[Sharathnasa]

@jackytu256 did you fixed this issue?

[meetps]

Check my comment here

[niuniu111]

@jackytu256 did you fixed this issue?