IndexError: Out of bounds on buffer access (axis 0) #85
Comments
|
I run it again and failed in the same 840 iterations |
|
I also meet the same error. Could you share how to address this problem? |
|
running into this as well @pkuCactus @PkuRainBow were you able to fix the problem? |
|
maybe you can see #36. it works for me. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,
Has any one encounter the problem that
IndexError: Out of bounds on buffer access (axis 0)?I'm using the Horovod with 16 GPUs for training and after 840 training iters, the training procedure failed. The logs are attached below,
bbox_loss=0.229750, mask_loss=0.375499, fcn_loss=0.201432, panoptic_accuracy=0.931710, panoptic_loss=0.195533,
INFO:root:Batch [840] Speed: 19.59 samples/sec Train-rpn_cls_loss=0.051714, rpn_bbox_loss=0.148433, rcnn_accuracy=0.889802, cls_loss=0.285857, bbox_loss=0.228536, mask_loss=0.373881, fcn_loss=0.201655, panoptic_accuracy=0.931608, panoptic_loss=0.196187,
2019-09-03 14:38:38,282 | callback.py | line 40 : Batch [840] Speed: 19.59 samples/sec Train-rpn_cls_loss=0.051714, rpn_bbox_loss=0.148433, rcnn_accuracy=0.889802, cls_loss=0.285857, bbox_loss=0.228536, mask_loss=0.373881, fcn_loss=0.201655, panoptic_accuracy=0.931608, panoptic_loss=0.196187,
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/bbox/bbox_transform.py:358: RuntimeWarning: invalid value encountered in log
targets_dw = ww * np.log(gt_widths / ex_widths)
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/bbox/bbox_transform.py:358: RuntimeWarning: invalid value encountered in log
targets_dw = ww * np.log(gt_widths / ex_widths)
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/bbox/bbox_transform.py:358: RuntimeWarning: invalid value encountered in log
targets_dw = ww * np.log(gt_widths / ex_widths)
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnet_upsnet.py", line 109, in forward
rois, _ = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnet_upsnet.py", line 109, in forward
rois, _ = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnet_upsnet.py", line 109, in forward
rois, _ = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnet_upsnet.py", line 109, in forward
rois, _ = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnet_upsnet.py", line 109, in forward
rois, _ = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnet_upsnet.py", line 109, in forward
rois, _ = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnet_upsnet.py", line 109, in forward
rois, _ = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnet_upsnet.py", line 109, in forward
rois, _ = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnet_upsnet.py", line 109, in forward
rois, _ = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnet_upsnet.py", line 109, in forward
rois, _ = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnet_upsnet.py", line 109, in forward
rois, _ = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnet_upsnet.py", line 109, in forward
rois, _ = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnet_upsnet.py", line 109, in forward
rois, _ = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnet_upsnet.py", line 109, in forward
rois, _ = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnet_upsnet.py", line 109, in forward
rois, _ = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnet_upsnet.py", line 109, in forward
rois, _ = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
The text was updated successfully, but these errors were encountered: