inner_product_test_float.TestsInnerProduct tests fail

[steenax86]

Hi,

While recently building and installing on a Haswell node (Intel(R) Core(TM) i7-5960X CPU @ 3.00GHz)
The following tests fail:

TestInnerProductForward/inner_product_test_float.TestsInnerProduct/0 and TestInnerProductForward/inner_product_test_float.TestsInnerProduct/1 fail on
[==========] 364 tests from 44 test cases ran. (396301 ms total)
[ PASSED ] 362 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] TestInnerProductForward/inner_product_test_float.TestsInnerProduct/0, where GetParam() = 44-byte object <40-00 00-00 01-00 00-00 05-00 00-00 09-00 00-00 03-00 00-00 04-00 00-00 02-00 00-00 20-00 00-00 30-00 00-00 06-00 00-00 06-00 00-00>
[ FAILED ] TestInnerProductForward/inner_product_test_float.TestsInnerProduct/1, where GetParam() = 44-byte object <40-00 00-00 01-00 00-00 07-00 00-00 07-00 00-00 03-00 00-00 04-00 00-00 02-00 00-00 20-00 00-00 30-00 00-00 06-00 00-00 06-00 00-00>

[emfomenk]

Hi @steenax86,

What compiler do you use? Is it gcc?
If this is the case I suspect the problem is in two OMP runtimes loaded (GNU because of the compiler and Intel because of using mklml library). Could you please try the following run line just to confirm the assumption?

# usually MKL_ML_LIB_PATH = external/mklml_lnx_$ver/lib
$ LD_LIBRARY_PATH=$MKL_ML_LIB_PATH:$LD_LIBRARY_PATH \
    LD_PRELOAD=libiomp5.so build/tests/gtests/tests_gtest

[steenax86]

Hello @emfomenk ,

Thank you for your response. I should have mentioned that I am using the Intel compiler suite, and here are the environment variables:

• echo $MKL_ML_LIB_PATH
  /home/smonteir/mkl-dnn-master/external/mklml_lnx_2017.0.0.20160801
• echo $LD_LIBRARY_PATH
  /home/smonteir/mkl-dnn-master/external/mklml_lnx_2017.0.0.20160801:/opt/intel/compilers_and_libraries_2016.2.181/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2016.2.181/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2016.2.181/linux/mpi/intel64/lib:/opt/intel/compilers_and_libraries_2016.2.181/linux/mpi/mic/lib:/opt/intel/compilers_and_libraries_2016.2.181/linux/ipp/lib/intel64:/opt/intel/compilers_and_libraries_2016.2.181/linux/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64:/opt/intel/compilers_and_libraries_2016.2.181/linux/tbb/lib/intel64/gcc4.4:/opt/intel/debugger_2016/libipt/intel64/lib:/opt/intel/compilers_and_libraries_2016.2.181/linux/daal/lib/intel64_lin:/opt/intel/compilers_and_libraries_2016.2.181/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/opt/intel/compilers_and_libraries_2016.2.181/linux/daal/../compiler/lib/intel64_lin:/opt/intel/vtune_amplifier_xe/bin64:
• ldd tests_gtest
  linux-vdso.so.1 => (0x00007ffeef7ef000)
  libmkldnn.so => /home/smonteir/mkl-dnn-master/src/libmkldnn.so (0x00007f5e7b769000)
  libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5e7b449000)
  libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5e7b143000)
  libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f5e7af33000)
  libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5e7ad1d000)
  libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5e7aaff000)
  libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5e7a739000)
  libmkl_rt.so => /opt/intel/compilers_and_libraries_2016.2.181/linux/mkl/lib/intel64/libmkl_rt.so (0x00007f5e7a172000)
  /lib64/ld-linux-x86-64.so.2 (0x0000564108a38000)
  libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5e79f6d000)

However...The error points to an incorrect evaluation in TestInnerProductForward/inner_product_test_float.TestsInnerProduct/1 listed in LastTest.log (from Testing/Temporary), which elaborates the failure message as

[----------] 4 tests from TestInnerProductForward/inner_product_test_float
[ RUN ] TestInnerProductForward/inner_product_test_float.TestsInnerProduct/0
[ OK ] TestInnerProductForward/inner_product_test_float.TestsInnerProduct/0 (27 ms)
[ RUN ] TestInnerProductForward/inner_product_test_float.TestsInnerProduct/1
/home/smonteir/mkl-dnn-master/tests/gtests/mkldnn_test_common.hpp:152: Failure
The difference between e and 0.0 is 0.83276450634002686, which exceeds 1e-4, where
e evaluates to 0.83276450634002686,
0.0 evaluates to 0, and
1e-4 evaluates to 0.0001.
/home/smonteir/mkl-dnn-master/tests/gtests/mkldnn_test_common.hpp:152: Failure
The difference between e and 0.0 is 0.83273166418075562, which exceeds 1e-4, where
e evaluates to 0.83273166418075562,
0.0 evaluates to 0, and
1e-4 evaluates to 0.0001........ (truncated)

Another oddity is that running ./tests_gtests still shows both TestInnerProductForward/inner_product_test_float.TestsInnerProduct/0 and TestInnerProductForward/inner_product_test_float.TestsInnerProduct/1 to fail. But make test
shows only one of them to fail.

Thanks,
Steena

[emfomenk]

Hi Steena,

I still see the dependency on libgomp.so in ldd tests_gtest:

libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f5e7af33000)

So could you please try using LD_PRELOAD trick just to make sure that this is not the case?
Using GNU and Intel omp runtime simultaneously is undefined behavior, so even running ./tests_gtests several times may lead to different results :)

[steenax86]

Hello Evarist (@emfomenk),

Thank you so much! export LD_PRELOAD=/libiomp5.so helped and all tests now pass---during both make tests and ./tests_gtest ! I still see the libgomp.so.1 dependency in tests_gtests, but it looks like threading in Intel MKL now links with libiomp5.so and not libgomp.so.1.

Thanks again,
Steena