Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Testcase failed during make runtest #1294

Closed
moegitree opened this issue Oct 16, 2014 · 15 comments
Closed

Testcase failed during make runtest #1294

moegitree opened this issue Oct 16, 2014 · 15 comments

Comments

@moegitree
Copy link

Hi, I am trying execute "make runtest" and met some problems.
The caffe is successfully compiled on Ubuntu12.04 with CPU only.
When executing the testcases, it shows like this:

src/caffe/test/test_math_functions.cpp:109: Failure
Value of: x[i] < 0 ? 1 : 0
Actual: 1
Expected: signbits[i]
Which is: 512
..................
[ PASSED ] 450 tests.
[ FAILED ] 7 tests, listed below:
[ FAILED ] MathFunctionsTest/0.TestSgnbitCPU, where TypeParam = float
[ FAILED ] MathFunctionsTest/1.TestSgnbitCPU, where TypeParam = double
[ FAILED ] MathFunctionsTest/1.TestHammingDistanceCPU, where TypeParam = double
[ FAILED ] EltwiseLayerTest/0.TestProd, where TypeParam = caffe::FloatCPU
[ FAILED ] EltwiseLayerTest/0.TestSum, where TypeParam = caffe::FloatCPU
[ FAILED ] EltwiseLayerTest/1.TestProd, where TypeParam = caffe::DoubleCPU
[ FAILED ] EltwiseLayerTest/1.TestSum, where TypeParam = caffe::DoubleCPU

I am a new learner of linux and caffe. Great thanks for any help.
@songjun54cm
Copy link

I have the similar problem:
[ FAILED ] 2 tests, listed below:
[ FAILED ] PoolingLayerTest/1.TestGPUGradientMax, where TypeParam = double
[ FAILED ] MathFunctionsTest/1.TestFabsGPU, where TypeParam = double

also seeking help now~~

@stas-sl
Copy link

stas-sl commented Nov 5, 2014

Same issue as for @moegitree. Did you find any causes?

@Yangqing
Copy link
Member

Yangqing commented Nov 6, 2014

The signbits bug should have been fixed by #1264 . Are you using the master branch or the dev branch? If you still encounter bugs after running the dev head, kindly let us know. Please attach the full make runtest result so we can see what went wrong.

@stas-sl
Copy link

stas-sl commented Nov 6, 2014

The full runtest log is here: https://gist.github.com/anonymous/1783e785315e63f380b4
After switching to dev branch there are only 5 test failures. Those with signbits are OK now.

It seems there are some precision issues:

[ RUN      ] EltwiseLayerTest/0.TestSum
src/caffe/test/test_eltwise_layer.cpp:102: Failure
Value of: in_data_a[i] + in_data_b[i] + in_data_c[i]
  Actual: 1.47906
Expected: data[i]
Which is: 1.47907

[ RUN      ] EltwiseLayerTest/0.TestProd
src/caffe/test/test_eltwise_layer.cpp:83: Failure
Value of: in_data_a[i] * in_data_b[i] * in_data_c[i]
  Actual: 0.00427637
Expected: data[i]
Which is: 0.00427638

Though in case of the following failures, I even can't find what's wrong, because expected and actual data seems to be equal. Or may be it is printed with some rounding.

[  FAILED  ] EltwiseLayerTest/1.TestProd, where TypeParam = caffe::DoubleCPU
[  FAILED  ] EltwiseLayerTest/1.TestSum, where TypeParam = caffe::DoubleCPU

Also something is wrong with caffe_cpu_hamming_distance:

[ RUN      ] MathFunctionsTest/1.TestHammingDistanceCPU
src/caffe/test/test_math_functions.cpp:79: Failure
Value of: caffe_cpu_hamming_distance<TypeParam>(n, x, y)
  Actual: 717829
Expected: this->ReferenceHammingDistance(n, x, y)
Which is: 1422821

I compiled caffe with CPU_ONLY := 1 and BLAS := open

@KTOC
Copy link

KTOC commented Nov 13, 2014

Hi,

I just want to update on the failed test, I got these 2 failures when run 'make runtest' :

[----------] 8 tests from CuDNNNeuronLayerTest/1, where TypeParam = double
[ RUN ] CuDNNNeuronLayerTest/1.TestSigmoidGradientCuDNN
[ OK ] CuDNNNeuronLayerTest/1.TestSigmoidGradientCuDNN (16 ms)
[ RUN ] CuDNNNeuronLayerTest/1.TestReLUGradientWithNegativeSlopeCuDNN
[ OK ] CuDNNNeuronLayerTest/1.TestReLUGradientWithNegativeSlopeCuDNN (20 ms)
[ RUN ] CuDNNNeuronLayerTest/1.TestTanHCuDNN
[ OK ] CuDNNNeuronLayerTest/1.TestTanHCuDNN (1 ms)
[ RUN ] CuDNNNeuronLayerTest/1.TestReLUCuDNN
[ OK ] CuDNNNeuronLayerTest/1.TestReLUCuDNN (0 ms)
[ RUN ] CuDNNNeuronLayerTest/1.TestReLUGradientCuDNN
[ OK ] CuDNNNeuronLayerTest/1.TestReLUGradientCuDNN (16 ms)
[ RUN ] CuDNNNeuronLayerTest/1.TestTanHGradientCuDNN
[ OK ] CuDNNNeuronLayerTest/1.TestTanHGradientCuDNN (22 ms)
[ RUN ] CuDNNNeuronLayerTest/1.TestReLUWithNegativeSlopeCuDNN
src/caffe/test/test_neuron_layer.cpp:455: Failure
Expected: (top_data[i]) >= (0.), actual: -0.0122354 vs 0
src/caffe/test/test_neuron_layer.cpp:456: Failure
Value of: top_data[i] == 0 || top_data[i] == bottom_data[i]
Actual: false
Expected: true
......

[----------] 8 tests from CuDNNNeuronLayerTest/0, where TypeParam = float
[ RUN ] CuDNNNeuronLayerTest/0.TestReLUCuDNN
[ OK ] CuDNNNeuronLayerTest/0.TestReLUCuDNN (0 ms)
[ RUN ] CuDNNNeuronLayerTest/0.TestReLUGradientWithNegativeSlopeCuDNN
[ OK ] CuDNNNeuronLayerTest/0.TestReLUGradientWithNegativeSlopeCuDNN (15 ms)
[ RUN ] CuDNNNeuronLayerTest/0.TestReLUWithNegativeSlopeCuDNN
src/caffe/test/test_neuron_layer.cpp:455: Failure
Expected: (top_data[i]) >= (0.), actual: -0.0122354 vs 0
src/caffe/test/test_neuron_layer.cpp:456: Failure
Value of: top_data[i] == 0 || top_data[i] == bottom_data[i]
Actual: false
Expected: true
....

The full log of these 2 errors is at http://pastebin.com/QD679cGj.

My system is Ubuntu 14.04, CUDA 6.5, cuDNN 6.5, opencv 2.4.9 and caffe dev branch

@Yangqing
Copy link
Member

OK, whoever wrote that TestReLUWithNegativeSlopeCuDNN function should be more careful... The test case was clearly wrong :(

I am going to fix it with a quick PR.

@Yangqing Yangqing mentioned this issue Nov 14, 2014
Merged
@Yangqing
Copy link
Member

Fixed.

@cmis91
Copy link

cmis91 commented Nov 18, 2014

exactly the same issue as observed in @stas-sl . the two sgnbit error disappear when switch to dev branch. ,@stas-sl, did you find he reason for the issue? BTW, I used Ubuntu 12, 32 bit. Is this cause the problem?

[----------] Global test environment tear-down
[==========] 457 tests from 98 test cases ran. (67848 ms total)
[ PASSED ] 450 tests.
[ FAILED ] 7 tests, listed below:
[ FAILED ] EltwiseLayerTest/0.TestProd, where TypeParam = caffe::FloatCPU
[ FAILED ] EltwiseLayerTest/0.TestSum, where TypeParam = caffe::FloatCPU
[ FAILED ] EltwiseLayerTest/1.TestProd, where TypeParam = caffe::DoubleCPU
[ FAILED ] EltwiseLayerTest/1.TestSum, where TypeParam = caffe::DoubleCPU
[ FAILED ] MathFunctionsTest/0.TestSgnbitCPU, where TypeParam = float
[ FAILED ] MathFunctionsTest/1.TestHammingDistanceCPU, where TypeParam = double
[ FAILED ] MathFunctionsTest/1.TestSgnbitCPU, where TypeParam = double

@Yangqing
Copy link
Member

Full log please? I think most likely it is some numerical precision problems.

@stas-sl
Copy link

stas-sl commented Nov 18, 2014

@cmis91, yes, I also have 32-bit Ubuntu. I didn't find the reason, but I didn't have any convergence issues because of that.

@ChengduoZhao
Copy link

hi, when i run make runtest on my computer (ubuntu12.04) ，I get this long

[ RUN ] MemoryDataLayerTest/0.TestSetBatchSize
OpenCV Error: Assertion failed (k == STD_VECTOR_MAT) in getMat, file /build/buildd/opencv-2.3.1/modules/core/src/matrix.cpp, line 918
unknown file: Failure
C++ exception with description "/build/buildd/opencv-2.3.1/modules/core/src/matrix.cpp:918: error: (-215) k == STD_VECTOR_MAT in function getMat
" thrown in the test body.
[ FAILED ] MemoryDataLayerTest/0.TestSetBatchSize, where TypeParam = caffe::FloatCPU (1 ms)
[ RUN ] MemoryDataLayerTest/0.AddMatVectorDefaultTransform
OpenCV Error: Assertion failed (k == STD_VECTOR_MAT) in getMat, file /build/buildd/opencv-2.3.1/modules/core/src/matrix.cpp, line 918
unknown file: Failure
C++ exception with description "/build/buildd/opencv-2.3.1/modules/core/src/matrix.cpp:918: error: (-215) k == STD_VECTOR_MAT in function getMat
" thrown in the test body.
[ FAILED ] MemoryDataLayerTest/0.AddMatVectorDefaultTransform, where TypeParam = caffe::FloatCPU (1 ms)
[----------] 5 tests from MemoryDataLayerTest/0 (21 ms total)

[----------] 3 tests from ThresholdLayerTest/1, where TypeParam = caffe::DoubleCPU

@engr3os engr3os mentioned this issue Jul 5, 2015
Closed
@mosellan
Copy link

mosellan commented Jul 8, 2015

Hi, I have the same problem that stas-si, when I do make runtest I've got this http://pastebin.com/KvubSdsP
and for eltwise_layer it's a little bit weird http://pastebin.com/kEjHmZ3p the values are exactly the same...
I'm using a 32 bits debian testing
Anyone can help me ? Thx !!

@LinZhineng
Copy link

I met some problems when trying execute "make runtest" . The caffe is compiled on Ubuntu12.04 with CPU only. Detail just listed below：

OpenCV Error: Assertion failed (k == STD_VECTOR_MAT) in getMat, file /build/buildd/opencv-2.3.1/modules/core/src/matrix.cpp, line 918
unknown file: Failure
C++ exception with description "/build/buildd/opencv-2.3.1/modules/core/src/matrix.cpp:918: error: (-215) k == STD_VECTOR_MAT in function getMat

[ FAILED ] 12 tests, listed below:
[ FAILED ] IOTest.TestDecodeDatumToCVMat
[ FAILED ] IOTest.TestDecodeDatumToCVMatContent
[ FAILED ] IOTest.TestDecodeDatumToCVMatContentNative
[ FAILED ] IOTest.TestDecodeDatum
[ FAILED ] IOTest.TestDecodeDatumToCVMatNativeGray
[ FAILED ] IOTest.TestDecodeDatumToCVMatNative
[ FAILED ] IOTest.TestDecodeDatumNative
[ FAILED ] IOTest.TestDecodeDatumNativeGray
[ FAILED ] MemoryDataLayerTest/0.AddMatVectorDefaultTransform, where TypeParam = caffe::FloatCPU
[ FAILED ] MemoryDataLayerTest/0.TestSetBatchSize, where TypeParam = caffe::FloatCPU
[ FAILED ] MemoryDataLayerTest/1.AddMatVectorDefaultTransform, where TypeParam = caffe::DoubleCPU
[ FAILED ] MemoryDataLayerTest/1.TestSetBatchSize, where TypeParam = caffe::DoubleCPU
12 FAILED TESTS
make: *** [runtest] Error 1
Many thanks for any help.

@LinZhineng
Copy link

@shuimulinxi
I met some problems when trying execute "make runtest" . The caffe is compiled on Ubuntu12.04 with CPU only.

OpenCV Error: Assertion failed (k == STD_VECTOR_MAT) in getMat, file /build/buildd/opencv-2.3.1/modules/core/src/matrix.cpp, line 918
unknown file: Failure
C++ exception with description "/build/buildd/opencv-2.3.1/modules/core/src/matrix.cpp:918: error: (-215) k == STD_VECTOR_MAT in function getMat

I chage the Makefile.config
PYTHON_LIB := /usr/lib "replace with" PYTHON_LIB := usr/local/lib

Succesful!!!

notes: when you compile once again, please execute "make clean" first

@cervantes-loves-ai
Copy link

[----------] 5 tests from ImageDataLayerTest/1, where TypeParam = caffe::CPUDevice
[ RUN ] ImageDataLayerTest/1.TestReshape
[ OK ] ImageDataLayerTest/1.TestReshape (46 ms)
[ RUN ] ImageDataLayerTest/1.TestShuffle
[ OK ] ImageDataLayerTest/1.TestShuffle (106 ms)
[ RUN ] ImageDataLayerTest/1.TestRead
[ OK ] ImageDataLayerTest/1.TestRead (103 ms)
[ RUN ] ImageDataLayerTest/1.TestResize
[ OK ] ImageDataLayerTest/1.TestResize (122 ms)
[ RUN ] ImageDataLayerTest/1.TestSpace
[ OK ] ImageDataLayerTest/1.TestSpace (63 ms)
[----------] 5 tests from ImageDataLayerTest/1 (440 ms total)

[----------] Global test environment tear-down
[==========] 1096 tests from 150 test cases ran. (68513 ms total)
[ PASSED ] 1095 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] NeuronLayerTest/0.TestPReLUForward, where TypeParam = caffe::CPUDevice

1 FAILED TEST
make[3]: *** [src/caffe/test/CMakeFiles/runtest] Error 1
make[2]: *** [src/caffe/test/CMakeFiles/runtest.dir/all] Error 2
make[1]: *** [src/caffe/test/CMakeFiles/runtest.dir/rule] Error 2
make: *** [runtest] Error 2

why this error?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@KTOC @Yangqing @cervantes-loves-ai @songjun54cm @ChengduoZhao @stas-sl @moegitree @cmis91 @LinZhineng @mosellan