Check failed: error == cudaSuccess (30 vs. 0) unknown error

[Coderx7]

Hello All,
I recently uninstalled my cuda 6.5 from my system since I had downloaded the newer version 7.5.
Before I uninstall the cuda 6.5 Caffe compiled just fine. now it doesnt.!
when It reaches the testall stage, it crashes with these errors:

'test_all.exe' (Win32): Loaded 'T:\Caffe\Build\x64\Release\test_all.exe'. Symbols loaded.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\ntdll.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\kernel32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\KernelBase.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\lmdb.dll'. Module was built without symbols.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\advapi32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\msvcrt.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\sechost.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\rpcrt4.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\msvcr120.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\opencv_imgproc2410.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\opencv_core2410.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\msvcp120.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\opencv_highgui2410.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\user32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\gdi32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\lpk.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\usp10.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\ole32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\oleaut32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\winsxs\amd64_microsoft.windows.common-controls_6595b64144ccf1df_5.82.7601.17514_none_a4d6a923711520a9\comctl32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\msvfw32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\winmm.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\shell32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\shlwapi.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\avifil32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\msacm32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\avicap32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\version.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\libopenblas.dll'. Module was built without symbols.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\libgfortran-3.dll'. Module was built without symbols.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\libquadmath-0.dll'. Module was built without symbols.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\libgcc_s_seh-1.dll'. Module was built without symbols.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\libglog.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\gflags.dll'. Module was built without symbols.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\cudnn64_4.dll'. Module was built without symbols.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\cublas64_75.dll'. Module was built without symbols.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\curand64_75.dll'. Module was built without symbols.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\cudart64_75.dll'. Module was built without symbols.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\hdf5.dll'. Symbols loaded.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\zlib.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\szip.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'D:4\Caffe\Build\x64\Release\hdf5_hl.dll'. Symbols loaded.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\python27.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\winsxs\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.6161_none_08e61857a83bc251\msvcr90.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\imm32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\msctf.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\nvcuda.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\ws2_32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\nsi.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\setupapi.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\cfgmgr32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\devobj.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\dwmapi.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Unloaded 'C:\Windows\System32\dwmapi.dll'
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\nvapi64.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\uxtheme.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\wintrust.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\crypt32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\msasn1.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Unloaded 'C:\Windows\System32\nvcuda.dll'
'test_all.exe' (Win32): Unloaded 'C:\Windows\System32\ws2_32.dll'
'test_all.exe' (Win32): Unloaded 'C:\Windows\System32\nsi.dll'
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\nvcuda.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\ws2_32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\nsi.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\dwmapi.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Unloaded 'C:\Windows\System32\dwmapi.dll'
'test_all.exe' (Win32): Unloaded 'C:\Windows\System32\nvcuda.dll'
'test_all.exe' (Win32): Unloaded 'C:\Windows\System32\ws2_32.dll'
'test_all.exe' (Win32): Unloaded 'C:\Windows\System32\nsi.dll'
E0411 01:38:28.942024 16012 common.cpp:122] Cannot create Cublas handle. Cublas won't be available.
E0411 01:38:28.965025 16012 common.cpp:129] Cannot create Curand generator. Curand won't be available.
F0411 01:38:28.966025 16012 syncedmem.hpp:18] Check failed: error == cudaSuccess (30 vs. 0)  unknown error
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\nvcuda.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\ws2_32.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\nsi.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Loaded 'C:\Windows\System32\dwmapi.dll'. Cannot find or open the PDB file.
'test_all.exe' (Win32): Unloaded 'C:\Windows\System32\dwmapi.dll'
'test_all.exe' (Win32): Unloaded 'C:\Windows\System32\nvcuda.dll'
'test_all.exe' (Win32): Unloaded 'C:\Windows\System32\ws2_32.dll'
'test_all.exe' (Win32): Unloaded 'C:\Windows\System32\nsi.dll'
The thread 0x1c84 has exited with code 3 (0x3).
The thread 0x3744 has exited with code 3 (0x3).
The program '[264] test_all.exe' has exited with code 3 (0x3).

Whats wrong?

[joelself]

cudaError_t is defined in driver_types.h in your Cuda include directory. It gives this super helpful description of error 30:

    /**
     * This indicates that an unknown internal error has occurred.
     */
    cudaErrorUnknown                      =     30,

Did you change your windows/CommonSettings.config to version 7.5?

    <PropertyGroup Label="UserMacros">
        <BuildDir>$(SolutionDir)..\Build</BuildDir>
        <!--NOTE: CpuOnlyBuild and UseCuDNN flags can't be set at the same time.-->
        <CpuOnlyBuild>false</CpuOnlyBuild>
        <UseCuDNN>true</UseCuDNN>
        <CudaVersion>7.5</CudaVersion>
        ...snip

[Coderx7]

yes, but It was fine untill I uninstalled cuda 6.5.
I tried installing it again, but nothing changed!

[willyd]

Can you confirm if the issue persists with the latest CMake build?

[Coderx7]

that issue was belongs to 6 7 month ago! I'm on win10 (the issue happened on win7 sp1)
nonetheless I check with the newer version and see if it compiles fine.
by the way is compiling straightforward and hassle-free?

[willyd]

I check with the newer version and see if it compiles fine.

Let me know how it goes.

by the way is compiling straightforward and hassle-free?

I tried to make it as easy as possible, but hassle-free is really a subjective thing. You can have a look at the appveyor build script to get an idea of how complicated it is.

[Coderx7]

@willyd : When I'm trying to run the cmake script which you provided in the readme, I get
The system cannot find the file specified.
this is the script :

set CMAKE_GENERATOR="Visual Studio 12 2013 Win64"
set CMAKE_CONFIGURATION=Release
mkdir build
cd build
cmake -G%CMAKE_GENERATOR% -DBLAS=Open -DCMAKE_BUILD_TYPE=%CMAKE_CONFIGURATION% -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX=<install_path> -C %CAFFE_DEPENDENCIES%\caffe-builder-config.cmake  ..\
cmake --build . --config %CMAKE_CONFIGURATION%
cmake --build . --config %CMAKE_CONFIGURATION% --target install

The visual studio 2013 solution file, works however!
am I missing something here?

[willyd]

@Coderx7 I am currently reworking the appveyor build script so it can be used outside of appveyor. You can see it there: https://github.com/willyd/caffe/tree/build-script

Building caffe with this should be as simple as running build_win.cmd

I will let you know when this is merged.

[Coderx7]

@willyd : I tried your new script, gotta say its much better now. Thank you for that.
How ever, I have not yet been able to have a successful compilation with it.
I get a cmake error :

CMake Error: The source directory "N:/caffe-cudnn5/build" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.
'-C' is not recognized as an internal or external command,
operable program or batch file.
ERROR: Configure failed

moving the cmakelists.txt from caffe_root to the build directory does not work either, it causes some more errors.

This is my config by the way : Link

I also have some suggestions , Instead of redownloading everything each time a simple error occures,
would it not be better to just use the previously downloaded files ? (use a cache for downloaded files, and only delete it when the whole compilation process was successful)
I think it would be a better idea to incorporate everything when compiling, meaning, e.g check to see if cmake exists, if it does, simple use it, if it does not, download it and use it locally (inside that cache folder I just suggested)

[willyd]

You forgot the line continuation character ^ on this line -DCUDNN_ROOT="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v8.0/cuda51/cuda"

[willyd]

I also have some suggestions , Instead of redownloading everything each time a simple error occures,
would it not be better to just use the previously downloaded files ? (use a cache for downloaded files, and only delete it when the whole compilation process was successful)
I think it would be a better idea to incorporate everything when compiling, meaning, e.g check to see if cmake exists, if it does, simple use it, if it does not, download it and use it locally (inside that cache folder I just suggested)

The build script was not really meant to be executed over and over. As documented in the README you should run it once and only once. After that you can edit the CMake Cache via cmake-gui.exe or by calling cmake.exe with new switches and building using VS or Ninja.

That being said, it would not be too hard to use CMake's execute_process together with ExternalProject to download and extract the dependencies on first CMake configure. If you are willing to submit a PR about that I can help you with that and merge it when it is ready.

[Coderx7]

@willyd : Thanks , those notes, solved all of my problems in my windows 10. every thing is just fine and works perfectly although I had to manually add dependencies to the path env.
But under windows 8.1 it gives a linker error ? ntdll.lib ? is this supposed to only work on windows 10? or am I missing something :

(By the way can you create a specific thread where these kinds of questions can be asked there? do you check Caffe-UserGroups? )

[willyd]

ntdll.lib is part of the Windows Driver Develpment kit. LMDB needs it so that we are able to create large database files without taking up your whole disk.

Now as for why it is not included in your install I guess that is an option of the compiler installer.

You can try to download the WDDK from microsoft or build on windows 10 and execute on windows 8.

(By the way can you create a specific thread where these kinds of questions can be asked there? do you check Caffe-UserGroups? )

Yes a receive a daily summary of caffe-users and try to give feedback to most windows requests. But specific threads for specific issues would be great indeed.

[willyd]

FYI ntdll.lib is located at C:\Program Files (x86)\Windows Kits\8.1\lib\winv6.3\um\x64 on my Windows 10 machine.

[willyd]

@Coderx7 Can we close this issue?

[Coderx7]

@willyd yeah sure