cudaSuccess (30 vs. 0) unknown error #4543
Comments
|
This is supposed to be due to an incorrect architecture set in the makefile. |
|
I have encounter the same error on my nvidia gtx1080 . |
|
@shashikale Here, fix it by (a) installing CUDA 8.0 and (b) change the caffe makefile.config.example and change the "50"s to "61" |
|
@shashikale have you ever finished the error? |
|
CUDA runtime error (30) might show if your program is unable to create or open the /dev/nvidia-uvm device file. This is usually fixed by installing package nvidia-modprobe:
thanks to @gheinrich |
|
Hi, how to solve the runtest error : BatchNormLayerTest/0.TestForward? I have the same error on Cuda 8.0, DeepLabv2, Titan Xp. I have tried several methods, but doesn't work. Help |
|
I solve this problem by add " sudo" command, i run "sudo make runtest" replace "make runtest", The problem is miss |
|
|
|
@gheinrich and @barnoncb , reboot is needed to do after |
Hi, did you manage to solve this issue at the end?? I'm having exactly the same problem. Thank you! |
|
@Smellly it is really right, though it is amazing |
I'm having issues getting 'make runtest' to complete. I thought maybe reducing batch size would help... but I'm not really sure what's causing the issues. Any help would be appreciated.
===========begin==============
.build_release/tools/caffe
libdc1394 error: Failed to initialize libdc1394
caffe: command line brew
usage: caffe
commands:
train train or finetune a model
test score a model
device_query show GPU diagnostic information
time benchmark model execution time
Flags from tools/caffe.cpp:
-gpu (Optional; run in GPU mode on given device IDs separated by ','.Use
'-gpu all' to run on all available GPUs. The effective training batch
size is multiplied by the number of devices.) type: string default: ""
-iterations (The number of iterations to run.) type: int32 default: 50
-model (The model definition protocol buffer text file.) type: string
default: ""
-sighup_effect (Optional; action to take when a SIGHUP signal is received:
snapshot, stop or none.) type: string default: "snapshot"
-sigint_effect (Optional; action to take when a SIGINT signal is received:
snapshot, stop or none.) type: string default: "stop"
-snapshot (Optional; the snapshot solver state to resume training.)
type: string default: ""
-solver (The solver definition protocol buffer text file.) type: string
default: ""
-weights (Optional; the pretrained weights to initialize finetuning,
separated by ','. Cannot be set simultaneously with snapshot.)
type: string default: ""
.build_release/test/test_all.testbin 0 #--gtest_shuffle
libdc1394 error: Failed to initialize libdc1394
Cuda number of devices: 0
Setting to use device 0
Current device id: 0
Current device name:
[==========] Running 2262 tests from 298 test cases.
[----------] Global test environment set-up.
[----------] 9 tests from AccuracyLayerTest/0, where TypeParam = float
[ RUN ] AccuracyLayerTest/0.TestSetup
E0730 19:33:33.571071 10661 common.cpp:113] Cannot create Cublas handle. Cublas won't be available.
E0730 19:33:33.573385 10661 common.cpp:120] Cannot create Curand generator. Curand won't be available.
[ OK ] AccuracyLayerTest/0.TestSetup (4 ms)
[ RUN ] AccuracyLayerTest/0.TestSetupTopK
[ OK ] AccuracyLayerTest/0.TestSetupTopK (1 ms)
[ RUN ] AccuracyLayerTest/0.TestSetupOutputPerClass
[ OK ] AccuracyLayerTest/0.TestSetupOutputPerClass (0 ms)
[ RUN ] AccuracyLayerTest/0.TestForwardCPU
[ OK ] AccuracyLayerTest/0.TestForwardCPU (0 ms)
[ RUN ] AccuracyLayerTest/0.TestForwardWithSpatialAxes
[ OK ] AccuracyLayerTest/0.TestForwardWithSpatialAxes (0 ms)
[ RUN ] AccuracyLayerTest/0.TestForwardIgnoreLabel
[ OK ] AccuracyLayerTest/0.TestForwardIgnoreLabel (0 ms)
[ RUN ] AccuracyLayerTest/0.TestForwardCPUTopK
[ OK ] AccuracyLayerTest/0.TestForwardCPUTopK (1 ms)
[ RUN ] AccuracyLayerTest/0.TestForwardCPUPerClass
[ OK ] AccuracyLayerTest/0.TestForwardCPUPerClass (1 ms)
[ RUN ] AccuracyLayerTest/0.TestForwardCPUPerClassWithIgnoreLabel
[ OK ] AccuracyLayerTest/0.TestForwardCPUPerClassWithIgnoreLabel (0 ms)
[----------] 9 tests from AccuracyLayerTest/0 (7 ms total)
[----------] 9 tests from AccuracyLayerTest/1, where TypeParam = double
[ RUN ] AccuracyLayerTest/1.TestSetup
[ OK ] AccuracyLayerTest/1.TestSetup (0 ms)
[ RUN ] AccuracyLayerTest/1.TestSetupTopK
[ OK ] AccuracyLayerTest/1.TestSetupTopK (0 ms)
[ RUN ] AccuracyLayerTest/1.TestSetupOutputPerClass
[ OK ] AccuracyLayerTest/1.TestSetupOutputPerClass (0 ms)
[ RUN ] AccuracyLayerTest/1.TestForwardCPU
[ OK ] AccuracyLayerTest/1.TestForwardCPU (0 ms)
[ RUN ] AccuracyLayerTest/1.TestForwardWithSpatialAxes
[ OK ] AccuracyLayerTest/1.TestForwardWithSpatialAxes (1 ms)
[ RUN ] AccuracyLayerTest/1.TestForwardIgnoreLabel
[ OK ] AccuracyLayerTest/1.TestForwardIgnoreLabel (0 ms)
[ RUN ] AccuracyLayerTest/1.TestForwardCPUTopK
[ OK ] AccuracyLayerTest/1.TestForwardCPUTopK (1 ms)
[ RUN ] AccuracyLayerTest/1.TestForwardCPUPerClass
[ OK ] AccuracyLayerTest/1.TestForwardCPUPerClass (0 ms)
[ RUN ] AccuracyLayerTest/1.TestForwardCPUPerClassWithIgnoreLabel
[ OK ] AccuracyLayerTest/1.TestForwardCPUPerClassWithIgnoreLabel (1 ms)
[----------] 9 tests from AccuracyLayerTest/1 (3 ms total)
[----------] 12 tests from ArgMaxLayerTest/0, where TypeParam = float
[ RUN ] ArgMaxLayerTest/0.TestSetup
E0730 19:33:33.579165 10661 common.cpp:140] Curand not available. Skipping setting the curand seed.
[ OK ] ArgMaxLayerTest/0.TestSetup (1 ms)
[ RUN ] ArgMaxLayerTest/0.TestSetupMaxVal
[ OK ] ArgMaxLayerTest/0.TestSetupMaxVal (2 ms)
[ RUN ] ArgMaxLayerTest/0.TestSetupAxis
[ OK ] ArgMaxLayerTest/0.TestSetupAxis (1 ms)
[ RUN ] ArgMaxLayerTest/0.TestSetupAxisNegativeIndexing
[ OK ] ArgMaxLayerTest/0.TestSetupAxisNegativeIndexing (2 ms)
[ RUN ] ArgMaxLayerTest/0.TestSetupAxisMaxVal
[ OK ] ArgMaxLayerTest/0.TestSetupAxisMaxVal (1 ms)
[ RUN ] ArgMaxLayerTest/0.TestCPU
[ OK ] ArgMaxLayerTest/0.TestCPU (2 ms)
[ RUN ] ArgMaxLayerTest/0.TestCPUMaxVal
[ OK ] ArgMaxLayerTest/0.TestCPUMaxVal (2 ms)
[ RUN ] ArgMaxLayerTest/0.TestCPUTopK
[ OK ] ArgMaxLayerTest/0.TestCPUTopK (2 ms)
[ RUN ] ArgMaxLayerTest/0.TestCPUMaxValTopK
[ OK ] ArgMaxLayerTest/0.TestCPUMaxValTopK (2 ms)
[ RUN ] ArgMaxLayerTest/0.TestCPUAxis
[ OK ] ArgMaxLayerTest/0.TestCPUAxis (8 ms)
[ RUN ] ArgMaxLayerTest/0.TestCPUAxisTopK
[ OK ] ArgMaxLayerTest/0.TestCPUAxisTopK (30 ms)
[ RUN ] ArgMaxLayerTest/0.TestCPUAxisMaxValTopK
[ OK ] ArgMaxLayerTest/0.TestCPUAxisMaxValTopK (28 ms)
[----------] 12 tests from ArgMaxLayerTest/0 (81 ms total)
[----------] 12 tests from ArgMaxLayerTest/1, where TypeParam = double
[ RUN ] ArgMaxLayerTest/1.TestSetup
[ OK ] ArgMaxLayerTest/1.TestSetup (4 ms)
[ RUN ] ArgMaxLayerTest/1.TestSetupMaxVal
[ OK ] ArgMaxLayerTest/1.TestSetupMaxVal (3 ms)
[ RUN ] ArgMaxLayerTest/1.TestSetupAxis
[ OK ] ArgMaxLayerTest/1.TestSetupAxis (3 ms)
[ RUN ] ArgMaxLayerTest/1.TestSetupAxisNegativeIndexing
[ OK ] ArgMaxLayerTest/1.TestSetupAxisNegativeIndexing (4 ms)
[ RUN ] ArgMaxLayerTest/1.TestSetupAxisMaxVal
[ OK ] ArgMaxLayerTest/1.TestSetupAxisMaxVal (3 ms)
[ RUN ] ArgMaxLayerTest/1.TestCPU
[ OK ] ArgMaxLayerTest/1.TestCPU (6 ms)
[ RUN ] ArgMaxLayerTest/1.TestCPUMaxVal
[ OK ] ArgMaxLayerTest/1.TestCPUMaxVal (5 ms)
[ RUN ] ArgMaxLayerTest/1.TestCPUTopK
[ OK ] ArgMaxLayerTest/1.TestCPUTopK (3 ms)
[ RUN ] ArgMaxLayerTest/1.TestCPUMaxValTopK
[ OK ] ArgMaxLayerTest/1.TestCPUMaxValTopK (4 ms)
[ RUN ] ArgMaxLayerTest/1.TestCPUAxis
[ OK ] ArgMaxLayerTest/1.TestCPUAxis (10 ms)
[ RUN ] ArgMaxLayerTest/1.TestCPUAxisTopK
[ OK ] ArgMaxLayerTest/1.TestCPUAxisTopK (32 ms)
[ RUN ] ArgMaxLayerTest/1.TestCPUAxisMaxValTopK
[ OK ] ArgMaxLayerTest/1.TestCPUAxisMaxValTopK (30 ms)
[----------] 12 tests from ArgMaxLayerTest/1 (108 ms total)
[----------] 3 tests from BatchNormLayerTest/0, where TypeParam = caffe::CPUDevice
[ RUN ] BatchNormLayerTest/0.TestForward
[ OK ] BatchNormLayerTest/0.TestForward (0 ms)
[ RUN ] BatchNormLayerTest/0.TestForwardInplace
[ OK ] BatchNormLayerTest/0.TestForwardInplace (1 ms)
[ RUN ] BatchNormLayerTest/0.TestGradient
[ OK ] BatchNormLayerTest/0.TestGradient (240 ms)
[----------] 3 tests from BatchNormLayerTest/0 (241 ms total)
[----------] 3 tests from BatchNormLayerTest/1, where TypeParam = caffe::CPUDevice
[ RUN ] BatchNormLayerTest/1.TestForward
[ OK ] BatchNormLayerTest/1.TestForward (1 ms)
[ RUN ] BatchNormLayerTest/1.TestForwardInplace
[ OK ] BatchNormLayerTest/1.TestForwardInplace (0 ms)
[ RUN ] BatchNormLayerTest/1.TestGradient
[ OK ] BatchNormLayerTest/1.TestGradient (239 ms)
[----------] 3 tests from BatchNormLayerTest/1 (240 ms total)
[----------] 3 tests from BatchNormLayerTest/2, where TypeParam = caffe::GPUDevice
[ RUN ] BatchNormLayerTest/2.TestForward
F0730 19:33:34.249238 10661 syncedmem.hpp:18] Check failed: error == cudaSuccess (30 vs. 0) unknown error
*** Check failure stack trace: ***
@ 0x2adc1c6e0daa (unknown)
@ 0x2adc1c6e0ce4 (unknown)
@ 0x2adc1c6e06e6 (unknown)
@ 0x2adc1c6e3687 (unknown)
@ 0x2adc1e859438 caffe::SyncedMemory::mutable_cpu_data()
@ 0x2adc1e8bbe5c caffe::Blob<>::Reshape()
@ 0x2adc1e8bc309 caffe::Blob<>::Reshape()
@ 0x2adc1e8bc39c caffe::Blob<>::Blob()
@ 0x4a26ab caffe::BatchNormLayerTest<>::BatchNormLayerTest()
@ 0x4a29cb testing::internal::TestFactoryImpl<>::CreateTest()
@ 0x99a773 testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x991375 testing::TestInfo::Run()
@ 0x991495 testing::TestCase::Run()
@ 0x9947d8 testing::internal::UnitTestImpl::RunAllTests()
@ 0x994a67 testing::UnitTest::Run()
@ 0x47314f main
@ 0x2adc1f834f45 (unknown)
@ 0x47bdc9 (unknown)
@ (nil) (unknown)
Aborted (core dumped)
make: *** [runtest] Error 134
The text was updated successfully, but these errors were encountered: