make runtest failed when I try to build caffe

[yechaochen]

I build caffe with CUDA7.5,OpenCV3.0,atlas,anaconda2,
The samples of CUDA named deviceQuery and bandwidthTest is passed. And I follow the step make, make test,make runtest with CPU only are successful.But something happened when I try to make in the GPU mode.

My Makefile.config:（something goes wrong when I use # ,so I replace it with %）

Refer to http://caffe.berkeleyvision.org/installation.html

% Contributions simplifying and improving our build system are welcome!
% cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 1
% CPU-only switch (uncomment to build without GPU support).
%CPU_ONLY := 1
% uncomment to disable IO dependencies and corresponding data layers
USE_OPENCV := 1
USE_LEVELDB := 1
USE_LMDB := 1
% uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
% You should not set this flag if you will be reading LMDBs with any
% possibility of simultaneous read and write
% ALLOW_LMDB_NOLOCK := 1
% Uncomment if you're using OpenCV 3
OPENCV_VERSION := 3
%To customize your choice of compiler, uncomment and set the following.
% N.B. the default for Linux is g++ and the default for OSX is clang++
% CUSTOM_CXX := g++
% CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda-7.5
% On Ubuntu 14.04, if cuda tools are installed via
% "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
% CUDA_DIR := /usr
% CUDA architecture setting: going with all of them.
% For CUDA < 6.0, comment the *50 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_20,code=sm_20
-gencode arch=compute_20,code=sm_21
-gencode arch=compute_30,code=sm_30
-gencode arch=compute_35,code=sm_35
-gencode arch=compute_50,code=sm_50
-gencode arch=compute_52,code=sm_52
-gencode arch=compute_50,code=compute_50
% BLAS choice:
% atlas for ATLAS (default)
% mkl for MKL
% open for OpenBlas
BLAS := atlas
% Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
% Leave commented to accept the defaults for your choice of BLAS
% (which should work)!
% BLAS_INCLUDE := /path/to/your/blas
% BLAS_LIB := /path/to/your/blas
% Homebrew puts openblas in a directory that is not on the standard search path
% BLAS_INCLUDE := $(shell brew --prefix openblas)/include
% BLAS_LIB := $(shell brew --prefix openblas)/lib
% This is required only if you will compile the matlab interface.
% MATLAB directory should contain the mex binary in /bin.
% MATLAB_DIR := /usr/local
% MATLAB_DIR := /Applications/MATLAB_R2012b.app
% NOTE: this is required only if you will compile the python interface.
% We need to be able to find Python.h and numpy/arrayobject.h.
%PYTHON_INCLUDE := /usr/include/python2.7
/usr/lib/python2.7/dist-packages/numpy/core/include
% Anaconda Python distribution is quite popular. Include path:
% Verify anaconda location, sometimes it's in root.
ANACONDA_HOME := /root/anaconda2
PYTHON_INCLUDE := $(ANACONDA_HOME)/include
$(ANACONDA_HOME)/include/python2.7
$(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include
% We need to be able to find libpythonX.X.so or .dylib.
%PYTHON_LIB := /usr/lib
PYTHON_LIB := $(ANACONDA_HOME)/lib
% Homebrew installs numpy in a non standard path (keg only)
% PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core._file)'))/include
% PYTHON_LIB += $(shell brew --prefix numpy)/lib
% Uncomment to support layers written in Python (will link against Python libs)
WITH_PYTHON_LAYER := 1
% Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib
% If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
% INCLUDE_DIRS += $(shell brew --prefix)/include
% LIBRARY_DIRS += $(shell brew --prefix)/lib
%Uncomment to use pkg-config to specify OpenCV library paths.
% (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
% USE_PKG_CONFIG := 1
BUILD_DIR := build
DISTRIBUTE_DIR := distribute
% Uncomment for debugging. Does not work on OSX due to #171
% DEBUG := 1
% The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0
% enable pretty build (comment to see full commands)
Q ?= @

When I make runtest a process named (DeconvolutionLayerTest/1.TestNDAgainst2D) is faild. Can someone help me to solve this problem. I have stucked for a week.

Update：
I try it again and the error like below is happened:
*src/caffe/test/test_common.cpp:58: Failure
Value of: ((const unsigned int)(data_b.cpu_data()))[i]
Actual: 1111099622
Expected: ((const unsigned int)(data_a.cpu_data()))[i]
Which is: 1199486454
src/caffe/test/test_common.cpp:58: Failure
Value of: ((const unsigned int)(data_b.cpu_data()))[i]
Actual: 1079127160
Expected: ((const unsigned int)(data_a.cpu_data()))[i]
Which is: 1082272888
[ FAILED ] CommonTest.TestRandSeedGPU (11001 ms)
*
WTF!!!! I quit the make runtest when I see the error.The information of error "TestNDAgainst2D" is similar to it.

[seanbell]

@shelhamer this test seems to be failing for a lot of people (https://github.com/BVLC/caffe/search?q=TestNDAgainst2D&type=Issues&utf8=%E2%9C%93).

I wonder if it's a numerical precision problem caused by different libraries being used by some people?

[seanbell]

Duplicate. See #4083.

[yechaochen]

@seanbell This problem just questioned in github few times. Does someting did I do was worng in the process building the caffe caused this error?

[seanbell]

I think this could be a bug in the unit test itself. I was just closing the issue because it's a duplicate. Please comment on #4083 to continue the discussion.

[siumk]

Not sure if it helps. My observation is that the test would fail if NV Digits server is running.