matcaffe throws a block of errors while running eval_all.m

[tkasarla]

I am on a shared cluster and not a root user.

While doing make matcaffe, I encountered a warning

Warning: You are using gcc version "5.4.0".  The version
         currently supported with MEX is "4.4.x".
         For a list of currently supported compilers see: 
         http://www.mathworks.com/support/compilers/current_release/

Since it was only a warning, I ignored it.

I added caffe path to matlab by addpath('/path/to/caffe/matlab')

I got the following error while trying to execute the file eval_all.m by ./run.sh

/usr/local/MATLAB/R2013a/bin/glnxa64/../../sys/os/glnxa64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by
/users/tejaswi.k/caffe_path/caffe/matlab/+caffe/private/caffe_.mexa64)

From https://github.com/BVLC/caffe/issues/827, I added this to .bashrc: export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6

And now, if I try to run ./run.sh I get another error as below

/users/tejaswi.k/caffe_path/caffe/matlab/+caffe/private/caffe_.mexa64: undefined symbol:
_ZN2cv8imencodeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_11_InputArrayERSt6vectorIhSaIhEERKSB_IiSaIiEE

From https://github.com/BVLC/caffe/issues/3934, I understood that it is issue with the caffe being built from system's opencv and matlab having its own opencv libraries.

There they suggested to change symbolic links in /bin/glnxa64, but since I am not a root user on the shared cluster, cannot change the symbolic links in /bin/glnxa64 .

I instead added following to .bashrc :

export LD_PRELOAD = /usr/lib/x86_64-linux-gnu/libopencv_[core,highgui,imgproc]

Now, it also requires libtiff.so.5 location for the /usr/lib of image processing libraries, so I added that too.

Doing this throws more errors and matlab force closes.

The errors are given below:

malloc: unknown:0: assertion botched
free: called with unallocated block argument
last command: (null)
Aborting...


                                                            < M A T L A B (R) >
                                                  Copyright 1984-2013 The MathWorks, Inc.
                                                    R2013a (8.1.0.604) 64-bit (glnxa64)
                                                             February 15, 2013
    
[libprotobuf ERROR google/protobuf/text_format.cc:274] Error parsing text-format caffe.NetParameter: 49:12: Message type "caffe.LayerParameter" has no field named "bn_param".
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0312 04:16:05.166391 26629 upgrade_proto.cpp:88] Check failed: ReadProtoFromTextFile(param_file, param) Failed to parse NetParameter file: prototxt/pspnet101_cityscapes_713.prototxt
*** Check failure stack trace: ***

 
------------------------------------------------------------------------
            Abort signal detected at Sun Mar 12 00:55:05 2017
------------------------------------------------------------------------

Configuration:
  Crash Decoding     : Disabled
  Current Visual     : None
  Default Encoding   : UTF-8
  GNU C Library      : 2.23 stable
  MATLAB Architecture: glnxa64
  MATLAB Root        : /usr/local/MATLAB/R2013a
  MATLAB Version     : 8.1.0.604 (R2013a)
  Operating System   : Linux 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x86_64
  Processor ID       : x86 Family 6 Model 42 Stepping 7, GenuineIntel
  Virtual Machine    : Java 1.6.0_17-b04 with Sun Microsystems Inc. Java HotSpot(TM) 64-Bit Server VM mixed mode
  Window System      : No active display

Fault Count: 1


Abnormal termination:
Abort signal

Register State (from fault):
  RAX = 0000000000000000  RBX = 00007f960059f420
  RCX = 00007f9698685428  RDX = 0000000000000006
  RSP = 00007f9672711158  RBP = 00007f9672711430
  RSI = 00000000000067d5  RDI = 000000000000677d

   R8 = 0000000000000081   R9 = 00007f960059f440
  R10 = 0000000000000008  R11 = 0000000000000206
  R12 = 00007f960059f480  R13 = 00000000000000b7
  R14 = 00007f960059f420  R15 = 00007f96005a6de0

  RIP = 00007f9698685428  EFL = 0000000000000206

   CS = 0033   FS = 0000   GS = 0000

Stack Trace (from fault):
[  0] 0x00007f9698685428                    /lib/x86_64-linux-gnu/libc.so.6+00218152 gsignal+00000056
[  1] 0x00007f969868702a                    /lib/x86_64-linux-gnu/libc.so.6+00225322 abort+00000362
[  2] 0x00007f960038ae49             /usr/lib/x86_64-linux-gnu/libglog.so.0+00040521
[  3] 0x00007f960038c5cd             /usr/lib/x86_64-linux-gnu/libglog.so.0+00046541
[  4] 0x00007f960038e433             /usr/lib/x86_64-linux-gnu/libglog.so.0+00054323 _ZN6google10LogMessage9SendToLogEv+00000643
[  5] 0x00007f960038c15b             /usr/lib/x86_64-linux-gnu/libglog.so.0+00045403 _ZN6google10LogMessage5FlushEv+00000187
[  6] 0x00007f960038ee1e             /usr/lib/x86_64-linux-gnu/libglog.so.0+00056862 _ZN6google15LogMessageFatalD2Ev+00000014
[  7] 0x00007f95fb71df51 /users/tejaswi.k/caffe_path/caffe/matlab/+caffe/private/caffe_.mexa64+00954193
[  8] 0x00007f95fb6f7292 /users/tejaswi.k/caffe_path/caffe/matlab/+caffe/private/caffe_.mexa64+00795282
[  9] 0x00007f95fb67c99a /users/tejaswi.k/caffe_path/caffe/matlab/+caffe/private/caffe_.mexa64+00293274
[ 10] 0x00007f95fb67cc16 /users/tejaswi.k/caffe_path/caffe/matlab/+caffe/private/caffe_.mexa64+00293910 mexFunction+00000169
[ 11] 0x00007f968c3d5f8a     /usr/local/MATLAB/R2013a/bin/glnxa64/libmex.so+00110474 mexRunMexFile+00000090
[ 12] 0x00007f968c3d20f9     /usr/local/MATLAB/R2013a/bin/glnxa64/libmex.so+00094457
[ 13] 0x00007f968c3d2f1c     /usr/local/MATLAB/R2013a/bin/glnxa64/libmex.so+00098076
[ 14] 0x00007f969a3196b2 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_dispatcher.so+00562866 _ZN8Mfh_file11dispatch_fhEiPP11mxArray_tagiS2_+00000594
[ 15] 0x00007f9699da3bf6 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+04262902
[ 16] 0x00007f9699da437a /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+04264826
[ 17] 0x00007f9699da4eea /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+04267754
[ 18] 0x00007f9699c07bbd /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02575293
[ 19] 0x00007f9699c33412 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02753554
[ 20] 0x00007f9699c3353f /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02753855
[ 21] 0x00007f9699d50500 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+03921152
[ 22] 0x00007f9699b69868 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01927272
[ 23] 0x00007f9699bd4e50 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02367056
[ 24] 0x00007f969a3196b2 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_dispatcher.so+00562866 _ZN8Mfh_file11dispatch_fhEiPP11mxArray_tagiS2_+00000594
[ 25] 0x00007f968c9ae53a  /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcos.so+01672506
[ 26] 0x00007f968c94f13a  /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcos.so+01282362
[ 27] 0x00007f968c94f3be  /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcos.so+01283006
[ 28] 0x00007f968c95112c  /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcos.so+01290540
[ 29] 0x00007f968c9bc246  /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcos.so+01729094
[ 30] 0x00007f968ca3bcd8  /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcos.so+02251992
[ 31] 0x00007f969a2cbaf8 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_dispatcher.so+00244472 _ZN13Mfh_MATLAB_fn11dispatch_fhEiPP11mxArray_tagiS2_+00000488
[ 32] 0x00007f9699bb7256 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02245206
[ 33] 0x00007f9699b67a86 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01919622
[ 34] 0x00007f9699b6c374 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01938292
[ 35] 0x00007f9699b68993 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01923475
[ 36] 0x00007f9699b69797 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01927063
[ 37] 0x00007f9699bd4e50 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02367056
[ 38] 0x00007f969a3196b2 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_dispatcher.so+00562866 _ZN8Mfh_file11dispatch_fhEiPP11mxArray_tagiS2_+00000594
[ 39] 0x00007f968c9b2a2f  /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcos.so+01690159
[ 40] 0x00007f968c94e5c4  /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcos.so+01279428
[ 41] 0x00007f968c94f0b9  /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcos.so+01282233
[ 42] 0x00007f968c94f3be  /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcos.so+01283006
[ 43] 0x00007f968c95068d  /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcos.so+01287821
[ 44] 0x00007f968c9507ad  /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcos.so+01288109
[ 45] 0x00007f968c950a4c  /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcos.so+01288780 _Z27omConstructObjectWithClientN4mcos9COSNameIDEiPPK11mxArray_tagPKNS_9COSClientE+00000476
[ 46] 0x00007f968c9bbb3d  /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcos.so+01727293
[ 47] 0x00007f968ca3d673  /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcos.so+02258547
[ 48] 0x00007f969a2cbaf8 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_dispatcher.so+00244472 _ZN13Mfh_MATLAB_fn11dispatch_fhEiPP11mxArray_tagiS2_+00000488
[ 49] 0x00007f9699bb7256 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02245206
[ 50] 0x00007f9699b67a86 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01919622
[ 51] 0x00007f9699b6c374 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01938292
[ 52] 0x00007f9699b68993 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01923475
[ 53] 0x00007f9699b69797 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01927063
[ 54] 0x00007f9699bd4e50 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02367056
[ 55] 0x00007f969a3196b2 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_dispatcher.so+00562866 _ZN8Mfh_file11dispatch_fhEiPP11mxArray_tagiS2_+00000594
[ 56] 0x00007f9699da3bf6 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+04262902
[ 57] 0x00007f9699da437a /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+04264826
[ 58] 0x00007f9699da4eea /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+04267754
[ 59] 0x00007f9699c07bbd /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02575293
[ 60] 0x00007f9699c33412 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02753554
[ 61] 0x00007f9699c3353f /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02753855
[ 62] 0x00007f9699d50500 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+03921152
[ 63] 0x00007f9699b6c8ac /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01939628
[ 64] 0x00007f9699b68993 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01923475
[ 65] 0x00007f9699b69797 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01927063
[ 66] 0x00007f9699bd4e50 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02367056
[ 67] 0x00007f969a3196b2 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_dispatcher.so+00562866 _ZN8Mfh_file11dispatch_fhEiPP11mxArray_tagiS2_+00000594
[ 68] 0x00007f9699da3bf6 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+04262902
[ 69] 0x00007f9699da437a /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+04264826
[ 70] 0x00007f9699da4eea /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+04267754
[ 71] 0x00007f9699c07bbd /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02575293
[ 72] 0x00007f9699c33412 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02753554
[ 73] 0x00007f9699c3353f /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02753855
[ 74] 0x00007f9699d50500 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+03921152
[ 75] 0x00007f9699b69868 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01927272
[ 76] 0x00007f9699bd4e50 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02367056
[ 77] 0x00007f969a3196b2 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_dispatcher.so+00562866 _ZN8Mfh_file11dispatch_fhEiPP11mxArray_tagiS2_+00000594
[ 78] 0x00007f9699ba3dcb /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+02166219
[ 79] 0x00007f9699b617cc /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01894348
[ 80] 0x00007f9699b5de1d /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01879581
[ 81] 0x00007f9699b5e255 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01880661
[ 82] 0x00007f9699b605d0 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwm_interpreter.so+01889744
[ 83] 0x00007f968d210f13   /usr/local/MATLAB/R2013a/bin/glnxa64/libmwiqm.so+03284755 _ZNK3iqm18InternalEvalPlugin24inEvalCmdWithLocalReturnERKSbItSt11char_traitsItESaItEEP15inWorkSpace_tag+00000147
[ 84] 0x00007f968d2118b8   /usr/local/MATLAB/R2013a/bin/glnxa64/libmwiqm.so+03287224 _ZN3iqm18InternalEvalPlugin7executeEP15inWorkSpace_tagRN5boost10shared_ptrIN14cmddistributor17IIPCompletedEventEEE+00000120
[ 85] 0x00007f969a5a3a15   /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcr.so+00391701
[ 86] 0x00007f968d1924fa   /usr/local/MATLAB/R2013a/bin/glnxa64/libmwiqm.so+02766074
[ 87] 0x00007f968d17fe24   /usr/local/MATLAB/R2013a/bin/glnxa64/libmwiqm.so+02690596
[ 88] 0x00007f968c5f93fd /usr/local/MATLAB/R2013a/bin/glnxa64/libmwbridge.so+00115709 _Z10ioReadLinebP8_IO_FILERKN5boost8optionalIKP15inWorkSpace_tagEEb+00000429
[ 89] 0x00007f968c5f9a84 /usr/local/MATLAB/R2013a/bin/glnxa64/libmwbridge.so+00117380
[ 90] 0x00007f968c5ff49d /usr/local/MATLAB/R2013a/bin/glnxa64/libmwbridge.so+00140445
[ 91] 0x00007f968c5ff59e /usr/local/MATLAB/R2013a/bin/glnxa64/libmwbridge.so+00140702
[ 92] 0x00007f968c5ffc7f /usr/local/MATLAB/R2013a/bin/glnxa64/libmwbridge.so+00142463 _Z8mnParserv+00000623
[ 93] 0x00007f969a5b13d2   /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcr.so+00447442 _ZN11mcrInstance30mnParser_on_interpreter_threadEv+00000034
[ 94] 0x00007f969a5909ac   /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcr.so+00313772
[ 95] 0x00007f969a590b88   /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcr.so+00314248
[ 96] 0x00007f969abd5cab /usr/local/MATLAB/R2013a/bin/glnxa64/libmwservices.so+01166507 _ZN10eventqueue18UserEventQueueImpl5flushEv+00000395
[ 97] 0x00007f9689b205fd   /usr/local/MATLAB/R2013a/bin/glnxa64/libmwuix.so+00534013
[ 98] 0x00007f969ac7ba9d /usr/local/MATLAB/R2013a/bin/glnxa64/libmwservices.so+01845917
[ 99] 0x00007f969ac7c50f /usr/local/MATLAB/R2013a/bin/glnxa64/libmwservices.so+01848591 _Z25svWS_ProcessPendingEventsiib+00001615
[100] 0x00007f969a5915ef   /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcr.so+00316911
[101] 0x00007f969a591f5c   /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcr.so+00319324
[102] 0x00007f969a58b592   /usr/local/MATLAB/R2013a/bin/glnxa64/libmwmcr.so+00292242
[103] 0x00007f9698a206ba              /lib/x86_64-linux-gnu/libpthread.so.0+00030394
[104] 0x00007f969875682d                    /lib/x86_64-linux-gnu/libc.so.6+01075245 clone+00000109


This error was detected while a MEX-file was running. If the MEX-file
is not an official MathWorks function, please examine its source code
for errors. Please consult the External Interfaces Guide for information
on debugging MEX-files.

If this problem is reproducible, please submit a Service Request via:
    http://www.mathworks.com/support/contact_us/

A technical support engineer might contact you with further information.

Thank you for your help.

Please help in resolving the issue.

[qizhuli]

I did almost the same thing as you, and got the following error as well in Matlab whenever I tried to carry out some system-level commands from Matlab (system(), ls, movefile(), copyfile(), etc...).

malloc: unknown:0: assertion botched
free: called with unallocated block argument
last command: (null)
Aborting...

The solution for me is Instead of preloading the libopencv libraries found in /usr/lib/x86_64-linux-gnu/, preload the ones in Anaconda.

[ghost]

I am also having the same issue. I don't have Anaconda. Could anyone refer the other solution. Thank you.

[tkasarla]

@dharma-kc, you could directly add opencv libraries path in the matlab file instead of export LD_PRELOAD = /usr/lib/x86_64-linux-gnu/libopencv_[core,highgui,imgproc].

In eval_all.m add path of the location where you opencv libraries are present:

addpath(/path/to/opencv/libs)

Let me know if it helps. It did for me.

[liaojiacai]

@tejaswikasarla it does‘t work,I have tried.