Gradient function not returning enough gradient

[Ume07]

https://github.com/pangyupo/mxnet_center_loss/
It works in mxnet 0.93. but it will be error in the new version
@piiswrong

Environment info

Operating System: Ubuntu 16.04

Package used (Python/R/Scala/Julia): python

MXNet version: 0.10.1/ 0.11.0

If you are using python package, please provide

Python version and distribution: Python 2.7.12

Error Message:

batchsize is 100
[16:07:10] src/io/iter_mnist.cc:112: MNISTIter: load 60000 images, shuffle=1, shape=(100,1,28,28)
[16:07:10] src/io/iter_mnist.cc:112: MNISTIter: load 10000 images, shuffle=1, shape=(100,1,28,28)
training model ...
dev is [gpu(0)]
/home/mxnet_center_loss/train_model.py:133: DeprecationWarning: mxnet.model.FeedForward has been deprecated. Please use mxnet.mod.Module instead.
**model_args)
/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/initializer.py:370: DeprecationWarning: Calling initializer with init(str, NDArray) has been deprecated.please use init(mx.init.InitDesc(...), NDArray) instead.
init(name, arr)
[16:07:11] /home/mxnet/dmlc-core/include/dmlc/./logging.h:308: [16:07:11] src/pass/gradient.cc:159: Check failed: (*rit)->inputs.size() == input_grads.size() (5 vs. 2) Gradient function not returning enough gradient

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f34139007cc]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(+0x2a996e0) [0x7f3415c466e0]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(ZNSt17_Function_handlerIFN4nnvm5GraphES1_EPS2_E9_M_invokeERKSt9_Any_dataOS1+0x111) [0x7f34146a0be1]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm11ApplyPassesENS_5GraphERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x32c) [0x7f3415c766dc]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm9ApplyPassENS_5GraphERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x3c9) [0x7f3414a81fe9]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm4pass8GradientENS_5GraphESt6vectorINS_9NodeEntryESaIS3_EES5_S5_St8functionIFS3_OS5_EES6_IFiRKNS_4NodeEEES6_IFS3_RKS3_SG_EES2_IPKNS_2OpESaISL_EENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x6fc) [0x7f3414b0bbbc]
[bt] (6) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet4exec13GraphExecutor13InitFullGraphEN4nnvm6SymbolERKSt6vectorINS_9OpReqTypeESaIS5_EE+0x863) [0x7f3414af3a43]
[bt] (7) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet4exec13GraphExecutor9InitGraphEN4nnvm6SymbolERKNS_7ContextERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES4_St4lessISD_ESaISt4pairIKSD_S4_EEERKSt6vectorIS4_SaIS4_EESR_SR_RKSN_INS_9OpReqTypeESaISS_EE+0x82) [0x7f3414af46d2]
[bt] (8) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet4exec13GraphExecutor4InitEN4nnvm6SymbolERKNS_7ContextERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES4_St4lessISD_ESaISt4pairIKSD_S4_EEERKSt6vectorINS_7NDArrayESaISO_EESS_RKSN_INS_9OpReqTypeESaIST_EESS_PNS_8ExecutorERKSt13unordered_mapINS2_9NodeEntryESO_NS2_13NodeEntryHashENS2_14NodeEntryEqualESaISG_IKS11_SO_EEE+0x8b7) [0x7f3414b00997]
[bt] (9) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(ZN5mxnet8Executor4BindEN4nnvm6SymbolERKNS_7ContextERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES3_St4lessISC_ESaISt4pairIKSC_S3_EEERKSt6vectorINS_7NDArrayESaISN_EESR_RKSM_INS_9OpReqTypeESaISS_EESR_PS0+0xe0) [0x7f3414b01d70]

Traceback (most recent call last):
File "train.py", line 98, in
main()
File "train.py", line 95, in main
train_model.fit(args, net, (train, val), data_shape)
File "/home/mxnet_center_loss/train_model.py", line 153, in fit
epoch_end_callback = checkpoint)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/model.py", line 847, in fit
sym_gen=self.sym_gen)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/model.py", line 227, in _train_multi_device
logger=logger)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/executor_manager.py", line 343, in init
self.slices, train_data)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/executor_manager.py", line 255, in init
input_types=data_types)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/executor_manager.py", line 201, in _bind_exec
grad_req=grad_req, shared_exec=base_exec)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/symbol/symbol.py", line 1661, in bind
ctypes.byref(handle)))
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/base.py", line 143, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [16:07:11] src/pass/gradient.cc:159: Check failed: (*rit)->inputs.size() == input_grads.size() (5 vs. 2) Gradient function not returning enough gradient

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f34139007cc]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(+0x2a996e0) [0x7f3415c466e0]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(ZNSt17_Function_handlerIFN4nnvm5GraphES1_EPS2_E9_M_invokeERKSt9_Any_dataOS1+0x111) [0x7f34146a0be1]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm11ApplyPassesENS_5GraphERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x32c) [0x7f3415c766dc]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm9ApplyPassENS_5GraphERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x3c9) [0x7f3414a81fe9]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm4pass8GradientENS_5GraphESt6vectorINS_9NodeEntryESaIS3_EES5_S5_St8functionIFS3_OS5_EES6_IFiRKNS_4NodeEEES6_IFS3_RKS3_SG_EES2_IPKNS_2OpESaISL_EENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x6fc) [0x7f3414b0bbbc]
[bt] (6) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet4exec13GraphExecutor13InitFullGraphEN4nnvm6SymbolERKSt6vectorINS_9OpReqTypeESaIS5_EE+0x863) [0x7f3414af3a43]
[bt] (7) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet4exec13GraphExecutor9InitGraphEN4nnvm6SymbolERKNS_7ContextERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES4_St4lessISD_ESaISt4pairIKSD_S4_EEERKSt6vectorIS4_SaIS4_EESR_SR_RKSN_INS_9OpReqTypeESaISS_EE+0x82) [0x7f3414af46d2]
[bt] (8) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet4exec13GraphExecutor4InitEN4nnvm6SymbolERKNS_7ContextERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES4_St4lessISD_ESaISt4pairIKSD_S4_EEERKSt6vectorINS_7NDArrayESaISO_EESS_RKSN_INS_9OpReqTypeESaIST_EESS_PNS_8ExecutorERKSt13unordered_mapINS2_9NodeEntryESO_NS2_13NodeEntryHashENS2_14NodeEntryEqualESaISG_IKS11_SO_EEE+0x8b7) [0x7f3414b00997]
[bt] (9) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(ZN5mxnet8Executor4BindEN4nnvm6SymbolERKNS_7ContextERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES3_St4lessISC_ESaISt4pairIKSC_S3_EEERKSt6vectorINS_7NDArrayESaISN_EESR_RKSM_INS_9OpReqTypeESaISS_EESR_PS0+0xe0) [0x7f3414b01d70]

Minimum reproducible example

https://github.com/pangyupo/mxnet_center_loss/

[huyangc]

Meet the same problem with the same custom layer

[tornadomeet]

Any update? same error here

[chenxu31]

I have had the same problem, any solution?

[stevenluzheng123456]

HI , did anyone solve this problem? it might by caused by custom layer, but I dont have any clue so fat

[szha]

@apache/mxnet-committers: This issue has been inactive for the past 90 days. It has no label and needs triage.

For general "how-to" questions, our user forum (and Chinese version) is a good place to get help.

[apeforest]

Ran the example in https://github.com/pangyupo/mxnet_center_loss/ and not able to reproduce the error any more. @sandeep-krishnamurthy Please help to close this issue.