possible bug in nd.gather_nd

[wshuail]

When I use nd.gather_nd with gpu, a unexpected error happened sometimes.

Here is the error message when I run my script with export MXNET_ENGINE_TYPE=NaiveEngine.

    offset = nd.gather_nd(offset_pred, fake_indices_0)
  File "<string>", line 55, in gather_nd
  File "/home/xxx/incubator-mxnet/python/mxnet/_ctypes/ndarray.py", line 92, in _imperative_invoke
    ctypes.byref(out_stypes)))
  File "/home/xxx/incubator-mxnet/python/mxnet/base.py", line 253, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [18:26:17] src/operator/tensor/./../../common/../operator/mxnet_op.h:794: Check failed: err == cuda$
uccess (77 vs. 0) : Name: mxnet_generic_kernel ErrStr:an illegal memory access was encountered
Stack trace:
  [bt] (0) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal($
+0x2b) [0x7f6eabe669cb]
  [bt] (1) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(void mxnet::op::mxnet_op::Kernel<mxnet::$
p::gather_nd, mshadow::gpu>::Launch<mxnet::OpReqType, int, int, int, mshadow::Shape<10>, float*, float*, float*>(mshadow::$
tream<mshadow::gpu>*, int, mxnet::OpReqType, int, int, int, mshadow::Shape<10>, float*, float*, float*)+0x1b0) [0x7f6eaed2$
d60]
  [bt] (2) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(void mxnet::op::GatherNDForward<mshadow:$
gpu>(nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std$
:vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob$
 > const&)+0x1e19) [0x7f6eaed36239]
  [bt] (3) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::imperative::PushFCompute(std::fun$
tion<void (nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const$
, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet:$
TBlob> > const&)> const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*,
std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const$
, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet$
:NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<unsigned int, std:$
allocator<unsigned int> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&)::{lambda(mxnet::
RunContext)#1}::operator()(mxnet::RunContext) const+0x372) [0x7f6eae355992] 

 [bt] (4) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(std::_Function_handler<void (mxnet::RunCo
ntext, mxnet::engine::CallbackOnComplete), mxnet::Engine::PushSync(std::function<void (mxnet::RunContext)>, mxnet::Context,
 std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::alloc
ator<mxnet::engine::Var*> > const&, mxnet::FnProperty, int, char const*)::{lambda(mxnet::RunContext, mxnet::engine::Callbac
kOnComplete)#1}>::_M_invoke(std::_Any_data const&, mxnet::RunContext, mxnet::engine::CallbackOnComplete)+0x74) [0x7f6eae9dd
fb4]
  [bt] (5) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::engine::NaiveEngine::PushAsync(std
::function<void (mxnet::RunContext, mxnet::engine::CallbackOnComplete)>, mxnet::Context, std::vector<mxnet::engine::Var*, s
td::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&,
 mxnet::FnProperty, int, char const*, bool)+0x167) [0x7f6eae9e3087]
  [bt] (6) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::Engine::PushSync(std::function<voi
d (mxnet::RunContext)>, mxnet::Context, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std:
:vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, mxnet::FnProperty, int, char const*)+0xf5) [0x7f
6eae9e09a5]
  [bt] (7) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::imperative::PushFCompute(std::func
tion<void (nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&
, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::
TBlob> > const&)> const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*,
std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&
, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet:
:NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<unsigned int, std::
allocator<unsigned int> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&)+0x295) [0x7f6eae
355405]
  [bt] (8) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::Imperative::InvokeOp(mxnet::Contex
t const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet:
:NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const
&, mxnet::DispatchMode, mxnet::OpStatePtr)+0x296) [0x7f6eae344a86]

terminate called after throwing an instance of 'dmlc::Error'
  what():  [18:26:17] src/storage/./pinned_memory_storage.h:73: CUDA: an illegal memory access was encountered
Stack trace:
  [bt] (0) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()
+0x2b) [0x7f6eabe669cb]
  [bt] (1) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::storage::PinnedMemoryStorage::Free
(mxnet::Storage::Handle)+0x92) [0x7f6eaea0c3a2]
  [bt] (2) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::StorageImpl::Free(mxnet::Storage::
Handle)+0x74) [0x7f6eaea0b304]
  [bt] (3) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(+0x370c119) [0x7f6eae3db119]
  [bt] (4) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(std::_Function_handler<void (mxnet::RunCo
ntext, mxnet::engine::CallbackOnComplete), mxnet::engine::NaiveEngine::DeleteVariable(std::function<void (mxnet::RunContext
)>, mxnet::Context, mxnet::engine::Var*)::{lambda(mxnet::RunContext, mxnet::engine::CallbackOnComplete)#1}>::_M_invoke(std:
:_Any_data const&, mxnet::RunContext, mxnet::engine::CallbackOnComplete)+0x7b) [0x7f6eae9e511b]
  [bt] (5) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::engine::NaiveEngine::PushAsync(std
::function<void (mxnet::RunContext, mxnet::engine::CallbackOnComplete)>, mxnet::Context, std::vector<mxnet::engine::Var*, s
td::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&,
 mxnet::FnProperty, int, char const*, bool)+0x216) [0x7f6eae9e3136]
  [bt] (6) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::engine::NaiveEngine::DeleteVariabl
e(std::function<void (mxnet::RunContext)>, mxnet::Context, mxnet::engine::Var*)+0x198) [0x7f6eae9e3d58]
  [bt] (7) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::NDArray::Chunk::~Chunk()+0x47d) [0
x7f6eae3d9ebd]
  [bt] (8) /home/xxx/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(std::_Sp_counted_base<(__gnu_cxx::_Lock_p
olicy)2>::_M_release()+0x3a) [0x7f6eabe6923a]

An error "terminate called without an active exception" could happen with the code below.

from mxnet import nd

ctx = mx.gpu(0)

for i in range(10000):
    if i % 100 == 0:
        print (i)

    batch_size = 2
    num_classes = 2
    width, height = 5, 5
    k = 3

    hm_origin = nd.random.uniform(0, 10, (batch_size, num_classes, width, height), ctx=ctx)

    hm = nd.reshape(hm_origin, (0, 0, -1))

    topk_scores, topk_idx = nd.topk(hm, k=k, ret_typ='both')

    topk_x_idx = nd.floor(topk_idx/width)
    topk_y_idx = (topk_idx%height)

    batch_idx = nd.repeat(nd.arange(batch_size), repeats=num_classes*k).reshape((1, -1))
    batch_idx = batch_idx.as_in_context(ctx)
    class_idx = nd.repeat(nd.arange(num_classes), repeats=batch_size*k).reshape((1, -1))
    class_idx = class_idx.as_in_context(ctx)

    topk_x_idx = nd.reshape(topk_x_idx, (1, -1))
    topk_y_idx = nd.reshape(topk_y_idx, (1, -1))

    indices = nd.concat(batch_idx, class_idx, topk_x_idx, topk_y_idx, dim=0)

    results = nd.gather_nd(hm_origin, indices)
    results = nd.reshape(results, (batch_size, num_classes, k))

when I add nd.waitall() at last, it works well.

any suggestions?

BTW, does mxnet has a function like torch.gather? gather_nd works close, but not so convenient.

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Bug

[frankfliu]

@mxnet-label-bot add [backend, cuda, question]