-
Notifications
You must be signed in to change notification settings - Fork 18.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cudaSuccess (30 vs. 0) unknown error #4543
Comments
This is supposed to be due to an incorrect architecture set in the makefile. |
I have encounter the same error on my nvidia gtx1080 . |
@shashikale Here, fix it by (a) installing CUDA 8.0 and (b) change the caffe makefile.config.example and change the "50"s to "61" |
@shashikale have you ever finished the error? |
CUDA runtime error (30) might show if your program is unable to create or open the /dev/nvidia-uvm device file. This is usually fixed by installing package nvidia-modprobe:
thanks to @gheinrich |
Hi, how to solve the runtest error : BatchNormLayerTest/0.TestForward? I have the same error on Cuda 8.0, DeepLabv2, Titan Xp. I have tried several methods, but doesn't work. Help |
I solve this problem by add " sudo" command, i run "sudo make runtest" replace "make runtest", The problem is miss |
|
@gheinrich and @barnoncb , reboot is needed to do after |
Hi, did you manage to solve this issue at the end?? I'm having exactly the same problem. Thank you! |
@Smellly it is really right, though it is amazing |
I'm having issues getting 'make runtest' to complete. I thought maybe reducing batch size would help... but I'm not really sure what's causing the issues. Any help would be appreciated.
===========begin==============
.build_release/tools/caffe
libdc1394 error: Failed to initialize libdc1394
caffe: command line brew
usage: caffe
commands:
train train or finetune a model
test score a model
device_query show GPU diagnostic information
time benchmark model execution time
Flags from tools/caffe.cpp:
-gpu (Optional; run in GPU mode on given device IDs separated by ','.Use
'-gpu all' to run on all available GPUs. The effective training batch
size is multiplied by the number of devices.) type: string default: ""
-iterations (The number of iterations to run.) type: int32 default: 50
-model (The model definition protocol buffer text file.) type: string
default: ""
-sighup_effect (Optional; action to take when a SIGHUP signal is received:
snapshot, stop or none.) type: string default: "snapshot"
-sigint_effect (Optional; action to take when a SIGINT signal is received:
snapshot, stop or none.) type: string default: "stop"
-snapshot (Optional; the snapshot solver state to resume training.)
type: string default: ""
-solver (The solver definition protocol buffer text file.) type: string
default: ""
-weights (Optional; the pretrained weights to initialize finetuning,
separated by ','. Cannot be set simultaneously with snapshot.)
type: string default: ""
.build_release/test/test_all.testbin 0 #--gtest_shuffle
libdc1394 error: Failed to initialize libdc1394
Cuda number of devices: 0
Setting to use device 0
Current device id: 0
Current device name:
[==========] Running 2262 tests from 298 test cases.
[----------] Global test environment set-up.
[----------] 9 tests from AccuracyLayerTest/0, where TypeParam = float
[ RUN ] AccuracyLayerTest/0.TestSetup
E0730 19:33:33.571071 10661 common.cpp:113] Cannot create Cublas handle. Cublas won't be available.
E0730 19:33:33.573385 10661 common.cpp:120] Cannot create Curand generator. Curand won't be available.
[ OK ] AccuracyLayerTest/0.TestSetup (4 ms)
[ RUN ] AccuracyLayerTest/0.TestSetupTopK
[ OK ] AccuracyLayerTest/0.TestSetupTopK (1 ms)
[ RUN ] AccuracyLayerTest/0.TestSetupOutputPerClass
[ OK ] AccuracyLayerTest/0.TestSetupOutputPerClass (0 ms)
[ RUN ] AccuracyLayerTest/0.TestForwardCPU
[ OK ] AccuracyLayerTest/0.TestForwardCPU (0 ms)
[ RUN ] AccuracyLayerTest/0.TestForwardWithSpatialAxes
[ OK ] AccuracyLayerTest/0.TestForwardWithSpatialAxes (0 ms)
[ RUN ] AccuracyLayerTest/0.TestForwardIgnoreLabel
[ OK ] AccuracyLayerTest/0.TestForwardIgnoreLabel (0 ms)
[ RUN ] AccuracyLayerTest/0.TestForwardCPUTopK
[ OK ] AccuracyLayerTest/0.TestForwardCPUTopK (1 ms)
[ RUN ] AccuracyLayerTest/0.TestForwardCPUPerClass
[ OK ] AccuracyLayerTest/0.TestForwardCPUPerClass (1 ms)
[ RUN ] AccuracyLayerTest/0.TestForwardCPUPerClassWithIgnoreLabel
[ OK ] AccuracyLayerTest/0.TestForwardCPUPerClassWithIgnoreLabel (0 ms)
[----------] 9 tests from AccuracyLayerTest/0 (7 ms total)
[----------] 9 tests from AccuracyLayerTest/1, where TypeParam = double
[ RUN ] AccuracyLayerTest/1.TestSetup
[ OK ] AccuracyLayerTest/1.TestSetup (0 ms)
[ RUN ] AccuracyLayerTest/1.TestSetupTopK
[ OK ] AccuracyLayerTest/1.TestSetupTopK (0 ms)
[ RUN ] AccuracyLayerTest/1.TestSetupOutputPerClass
[ OK ] AccuracyLayerTest/1.TestSetupOutputPerClass (0 ms)
[ RUN ] AccuracyLayerTest/1.TestForwardCPU
[ OK ] AccuracyLayerTest/1.TestForwardCPU (0 ms)
[ RUN ] AccuracyLayerTest/1.TestForwardWithSpatialAxes
[ OK ] AccuracyLayerTest/1.TestForwardWithSpatialAxes (1 ms)
[ RUN ] AccuracyLayerTest/1.TestForwardIgnoreLabel
[ OK ] AccuracyLayerTest/1.TestForwardIgnoreLabel (0 ms)
[ RUN ] AccuracyLayerTest/1.TestForwardCPUTopK
[ OK ] AccuracyLayerTest/1.TestForwardCPUTopK (1 ms)
[ RUN ] AccuracyLayerTest/1.TestForwardCPUPerClass
[ OK ] AccuracyLayerTest/1.TestForwardCPUPerClass (0 ms)
[ RUN ] AccuracyLayerTest/1.TestForwardCPUPerClassWithIgnoreLabel
[ OK ] AccuracyLayerTest/1.TestForwardCPUPerClassWithIgnoreLabel (1 ms)
[----------] 9 tests from AccuracyLayerTest/1 (3 ms total)
[----------] 12 tests from ArgMaxLayerTest/0, where TypeParam = float
[ RUN ] ArgMaxLayerTest/0.TestSetup
E0730 19:33:33.579165 10661 common.cpp:140] Curand not available. Skipping setting the curand seed.
[ OK ] ArgMaxLayerTest/0.TestSetup (1 ms)
[ RUN ] ArgMaxLayerTest/0.TestSetupMaxVal
[ OK ] ArgMaxLayerTest/0.TestSetupMaxVal (2 ms)
[ RUN ] ArgMaxLayerTest/0.TestSetupAxis
[ OK ] ArgMaxLayerTest/0.TestSetupAxis (1 ms)
[ RUN ] ArgMaxLayerTest/0.TestSetupAxisNegativeIndexing
[ OK ] ArgMaxLayerTest/0.TestSetupAxisNegativeIndexing (2 ms)
[ RUN ] ArgMaxLayerTest/0.TestSetupAxisMaxVal
[ OK ] ArgMaxLayerTest/0.TestSetupAxisMaxVal (1 ms)
[ RUN ] ArgMaxLayerTest/0.TestCPU
[ OK ] ArgMaxLayerTest/0.TestCPU (2 ms)
[ RUN ] ArgMaxLayerTest/0.TestCPUMaxVal
[ OK ] ArgMaxLayerTest/0.TestCPUMaxVal (2 ms)
[ RUN ] ArgMaxLayerTest/0.TestCPUTopK
[ OK ] ArgMaxLayerTest/0.TestCPUTopK (2 ms)
[ RUN ] ArgMaxLayerTest/0.TestCPUMaxValTopK
[ OK ] ArgMaxLayerTest/0.TestCPUMaxValTopK (2 ms)
[ RUN ] ArgMaxLayerTest/0.TestCPUAxis
[ OK ] ArgMaxLayerTest/0.TestCPUAxis (8 ms)
[ RUN ] ArgMaxLayerTest/0.TestCPUAxisTopK
[ OK ] ArgMaxLayerTest/0.TestCPUAxisTopK (30 ms)
[ RUN ] ArgMaxLayerTest/0.TestCPUAxisMaxValTopK
[ OK ] ArgMaxLayerTest/0.TestCPUAxisMaxValTopK (28 ms)
[----------] 12 tests from ArgMaxLayerTest/0 (81 ms total)
[----------] 12 tests from ArgMaxLayerTest/1, where TypeParam = double
[ RUN ] ArgMaxLayerTest/1.TestSetup
[ OK ] ArgMaxLayerTest/1.TestSetup (4 ms)
[ RUN ] ArgMaxLayerTest/1.TestSetupMaxVal
[ OK ] ArgMaxLayerTest/1.TestSetupMaxVal (3 ms)
[ RUN ] ArgMaxLayerTest/1.TestSetupAxis
[ OK ] ArgMaxLayerTest/1.TestSetupAxis (3 ms)
[ RUN ] ArgMaxLayerTest/1.TestSetupAxisNegativeIndexing
[ OK ] ArgMaxLayerTest/1.TestSetupAxisNegativeIndexing (4 ms)
[ RUN ] ArgMaxLayerTest/1.TestSetupAxisMaxVal
[ OK ] ArgMaxLayerTest/1.TestSetupAxisMaxVal (3 ms)
[ RUN ] ArgMaxLayerTest/1.TestCPU
[ OK ] ArgMaxLayerTest/1.TestCPU (6 ms)
[ RUN ] ArgMaxLayerTest/1.TestCPUMaxVal
[ OK ] ArgMaxLayerTest/1.TestCPUMaxVal (5 ms)
[ RUN ] ArgMaxLayerTest/1.TestCPUTopK
[ OK ] ArgMaxLayerTest/1.TestCPUTopK (3 ms)
[ RUN ] ArgMaxLayerTest/1.TestCPUMaxValTopK
[ OK ] ArgMaxLayerTest/1.TestCPUMaxValTopK (4 ms)
[ RUN ] ArgMaxLayerTest/1.TestCPUAxis
[ OK ] ArgMaxLayerTest/1.TestCPUAxis (10 ms)
[ RUN ] ArgMaxLayerTest/1.TestCPUAxisTopK
[ OK ] ArgMaxLayerTest/1.TestCPUAxisTopK (32 ms)
[ RUN ] ArgMaxLayerTest/1.TestCPUAxisMaxValTopK
[ OK ] ArgMaxLayerTest/1.TestCPUAxisMaxValTopK (30 ms)
[----------] 12 tests from ArgMaxLayerTest/1 (108 ms total)
[----------] 3 tests from BatchNormLayerTest/0, where TypeParam = caffe::CPUDevice
[ RUN ] BatchNormLayerTest/0.TestForward
[ OK ] BatchNormLayerTest/0.TestForward (0 ms)
[ RUN ] BatchNormLayerTest/0.TestForwardInplace
[ OK ] BatchNormLayerTest/0.TestForwardInplace (1 ms)
[ RUN ] BatchNormLayerTest/0.TestGradient
[ OK ] BatchNormLayerTest/0.TestGradient (240 ms)
[----------] 3 tests from BatchNormLayerTest/0 (241 ms total)
[----------] 3 tests from BatchNormLayerTest/1, where TypeParam = caffe::CPUDevice
[ RUN ] BatchNormLayerTest/1.TestForward
[ OK ] BatchNormLayerTest/1.TestForward (1 ms)
[ RUN ] BatchNormLayerTest/1.TestForwardInplace
[ OK ] BatchNormLayerTest/1.TestForwardInplace (0 ms)
[ RUN ] BatchNormLayerTest/1.TestGradient
[ OK ] BatchNormLayerTest/1.TestGradient (239 ms)
[----------] 3 tests from BatchNormLayerTest/1 (240 ms total)
[----------] 3 tests from BatchNormLayerTest/2, where TypeParam = caffe::GPUDevice
[ RUN ] BatchNormLayerTest/2.TestForward
F0730 19:33:34.249238 10661 syncedmem.hpp:18] Check failed: error == cudaSuccess (30 vs. 0) unknown error
*** Check failure stack trace: ***
@ 0x2adc1c6e0daa (unknown)
@ 0x2adc1c6e0ce4 (unknown)
@ 0x2adc1c6e06e6 (unknown)
@ 0x2adc1c6e3687 (unknown)
@ 0x2adc1e859438 caffe::SyncedMemory::mutable_cpu_data()
@ 0x2adc1e8bbe5c caffe::Blob<>::Reshape()
@ 0x2adc1e8bc309 caffe::Blob<>::Reshape()
@ 0x2adc1e8bc39c caffe::Blob<>::Blob()
@ 0x4a26ab caffe::BatchNormLayerTest<>::BatchNormLayerTest()
@ 0x4a29cb testing::internal::TestFactoryImpl<>::CreateTest()
@ 0x99a773 testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x991375 testing::TestInfo::Run()
@ 0x991495 testing::TestCase::Run()
@ 0x9947d8 testing::internal::UnitTestImpl::RunAllTests()
@ 0x994a67 testing::UnitTest::Run()
@ 0x47314f main
@ 0x2adc1f834f45 (unknown)
@ 0x47bdc9 (unknown)
@ (nil) (unknown)
Aborted (core dumped)
make: *** [runtest] Error 134
The text was updated successfully, but these errors were encountered: