Confusing behavior of some evaluation metrics

[asitstands]

mse = mx.metric.create('mse')
x = mx.nd.array([[1,2,3,4]])
mse.update(x, x + 0.1)
print(mse.get()) # 2.509999 ...? I expected 0.01.

According to the API document, the update method of evaluation metrics takes lists of NDArrays. So the first example uses the API not correctly, but there is no errors or warnings. Many functions of mxnet allow "an NDArray or a list of NDArrays" as an argument and an NDArray is equivalent to a list with a single element. So this behavior of the MSE metric is confusing and errorprone. Please raise an error or fix the implementation for NDArray arguments.

When I give NDArrays as arguments instead of lists to call Accuracy.update, a segfault occurs with the following error and stacktrace.

terminate called after throwing an instance of 'dmlc::Error'
  what():  [12:32:46] src/engine/./threaded_engine.h:359: [12:32:46] src/operator/tensor/broadcast_reduce_op.h:121: Check failed: axis < ndim && axis >= -ndim axis 1 exceeds the input dimension of 1

Stack trace returned 10 entries:
[bt] (0) /usr/lib/python3.6/site-packages/mxnet/libmxnet.so(dmlc::StackTrace[abi:cxx11]()+0x46) [0x7f385e63a4d6]
[bt] (1) /usr/lib/python3.6/site-packages/mxnet/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x28) [0x7f385e638308]
[bt] (2) /usr/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::op::CheckAxis(int, int)+0xc2) [0x7f385ea731b2]
[bt] (3) /usr/lib/python3.6/site-packages/mxnet/libmxnet.so(void mxnet::op::SearchAxisCompute<mshadow::cpu, mshadow::red::maximum>(nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)+0xa4) [0x7f385ec693d4]
[bt] (4) /usr/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::imperative::PushFCompute(std::function<void (nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)> const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&)::{lambda(mxnet::RunContext)#1}::operator()(mxnet::RunContext) const+0x330) [0x7f385fded110]
[bt] (5) /usr/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x212c4e9) [0x7f38601af4e9]
[bt] (6) /usr/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext, mxnet::engine::OprBlock*)+0xdd) [0x7f38601acb9d]
[bt] (7) /usr/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock*, bool)::{lambda()#1}::operator()() const::{lambda(std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>)#1}::operator()(mxnet::engine::ThreadPool::SimpleEvent) const+0xc1) [0x7f38601b94a1]
[bt] (8) /usr/lib/python3.6/site-packages/mxnet/libmxnet.so(std::_Function_handler<void (std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>), mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock*, bool)::{lambda()#1}::operator()() const::{lambda(std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>&&)+0x36) [0x7f38601b9266]
[bt] (9) /usr/lib/python3.6/site-packages/mxnet/libmxnet.so(void std::__invoke_impl<void, std::function<void (std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>)>, std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent> >(std::__invoke_other, std::function<void (std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>)>&&, std::shared_ptr<mxnet::engine::ThreadPool::SimpleEvent>&&)+0x3c) [0x7f38601b66fc]

This is much more acceptable than meaningless results without any error but not graceful.

mse = mx.metric.create('mse')
x = mx.nd.array([1,2,3,4]) 
mse.update([x], [x + 0.1]) # lists of 1D arrays.
print(mse.get()) # 2.509999 ... ?

The result of this example is not understandable. I believe the result should be 0.01 in this case also. Or, if 1D array is not acceptable, it must raise an error.

MAE, RMSE and PearsonCorrelation also show similar behavior.

[sxjscience]

Thanks for reporting this, we will improve our documentation and support NDArray as the input.

[sxjscience]

I find this is actually a bug. The pred will be reshaped if it has ndim=1, see https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/metric.py#L911-L912. In this case, the pred and the label have both shape (4,) and after reshaping, pred.shape will be changed to (4,1). pred - label will then generate an array with shape (4, 4) and give the incorrect output.