MKLBatchNormLayer segfaults on CIFAR10 test

[sorinescu]

Running examples/cifar10/train_full_sigmoid_bn.sh results in a SEGFAULT. See test instructions. The test was done on the CPU instead of GPU (see Why train on a GPU? in the instructions).

Final part of the output:

I1115 10:17:46.048431 16368 net.cpp:345] Network initialization done.
I1115 10:17:46.048583 16368 solver.cpp:104] Solver scaffolding done.
*** Aborted at 1479205066 (unix time) try "date -d @1479205066" if you are using GNU date ***
PC: @     0x7f4d4ab46869 caffe::Blob<>::Reshape()
*** SIGSEGV (@0x28) received by PID 16368 (TID 0x7f4d4b998a40) from PID 40; stack trace: ***
    @     0x7f4d46a71330 (unknown)
    @     0x7f4d4ab46869 caffe::Blob<>::Reshape()
    @     0x7f4d4ab474f5 caffe::Blob<>::Blob()
    @     0x7f4d4aa9930d caffe::Creator_SGDSolver<>()
    @     0x7f4d4b7bf707 caffe::SolverRegistry<>::CreateSolver()
    @     0x7f4d4b7aeaab train()
    @     0x7f4d4b7ad166 main
    @     0x7f4d466bdf45 (unknown)
    @     0x7f4d4b7acf69 (unknown)
    @                0x0 (unknown)
Segmentation fault

If I modify layer_factory.cpp and force the use of the Caffe implementation of the BatchNorm layer, it works:

  if (engine == BatchNormParameter_Engine_DEFAULT) {
//#if defined(USE_MKL2017_AS_DEFAULT_ENGINE)
//    engine = BatchNormParameter_Engine_MKL2017;
//#elif defined(USE_MKLDNN_AS_DEFAULT_ENGINE)
//    engine = BatchNormParameter_Engine_MKLDNN;
//#else
    engine = BatchNormParameter_Engine_CAFFE;
//#endif
  }

Relevant Makefile.config vars:

CPU_ONLY := 1
USE_MKL2017_AS_DEFAULT_ENGINE := 1
CUSTOM_CXX := icpc # 2017 update1
BLAS := mkl
WITH_PYTHON_LAYER := 1

[jczaja]

Thanks for letting us know. We will look at it shortly. Does the same problem occurs when caffe is build via GCC ?

[sorinescu]

It's built using the Intel compiler:
icpc (ICC) 17.0.1 20161005

[pnoga]

New version of MKL has been released as part of Intel Caffe. I recommend to check it out.