RuntimeError on my Ubuntu16.04 with CUDA.8.0 and cuDNN5.1 #5206
Comments
|
Delete your Theano cache. theano-cache purge If the path is set correctly. Le 9 nov. 2016 20:58, "xiaoiker" notifications@github.com a écrit :
|
|
No news, so closing. It work in jenkins on ubuntu 16.04, cuda 8 and cudnn 5.1 |
|
The failure to create a cudnn handle can be caused by lack of memory. During the call to cudnnCreate, as per my computations, about 20 MB are required. If theano.gpuarray.preallocate is too big, the cudnn init call fails with CUDNN_STATUS_INTERNAL_ERROR (which is not even documented in cudnn's user manual as being possible to return) or CUDNN_STATUS_NOT_INITIALIZED. So, maybe, if it's possible, it would be better if theano.gpuarray.init called dnn._make_handle before allocating with pygpu.empty.gmem, or, at least make sure that there are 20 MB free before dnn handle init. Also, as a sidenote, the preallocation size is not taken into consideration if it is specified via theano.config after theano is imported at runtime, and doesn't complain about it, either, as it happens with e.g. cnmem. |
|
@abergeron should look at changing the order of init.
thanks for that good diagnostic!
…On Tue, Nov 29, 2016 at 12:45 PM, bbudescu ***@***.***> wrote:
The failure to create a cudnn handle can be caused by lack of memory.
During the call to cudnnCreate, as per my computations, about 20 MB are
required. If theano.gpuarray.preallocate is too big, the cudnn init call
fails with CUDNN_STATUS_INTERNAL_ERROR (which is not even documented in
cudnn's user manual as being possible to return) or
CUDNN_STATUS_NOT_INITIALIZED. So, maybe, if it's possible, it would be
better if theano.gpuarray.*init* called dnn._make_handle before
allocating with pygpu.empty.gmem, or, at least make sure that there are 20
MB free before dnn handle init.
Also, as a sidenote, the preallocation size is not taken into
consideration if it is specified via theano.config after theano is imported
at runtime, and doesn't complain about it, either, as it happens with e.g.
cnmem.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<#5206 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AALC-5xmMmvsudFlfyw4EbpaJaypDGS2ks5rDGSsgaJpZM4KuMs9>
.
|
Traceback (most recent call last):
File "/home/animation/mProject/motionsynth_code/synth/train.py", line 57, in
trainer.train(network, E, E, filename='network_core.npz')
File "../nn/AdamTrainer.py", line 66, in train
}, allow_input_downcast=True)
File "/home/animation/myLib/Theano/theano/compile/function.py", line 326, in function
output_keys=output_keys)
File "/home/animation/myLib/Theano/theano/compile/pfunc.py", line 486, in pfunc
output_keys=output_keys)
File "/home/animation/myLib/Theano/theano/compile/function_module.py", line 1784, in orig_function
defaults)
File "/home/animation/myLib/Theano/theano/compile/function_module.py", line 1651, in create
input_storage=input_storage_lists, storage_map=storage_map)
File "/home/animation/myLib/Theano/theano/gof/link.py", line 699, in make_thunk
storage_map=storage_map)[:3]
File "/home/animation/myLib/Theano/theano/gof/vm.py", line 1057, in make_all
impl=impl))
File "/home/animation/myLib/Theano/theano/gof/op.py", line 924, in make_thunk
no_recycling)
File "/home/animation/myLib/Theano/theano/gof/op.py", line 828, in make_c_thunk
output_storage=node_output_storage)
File "/home/animation/myLib/Theano/theano/gof/cc.py", line 1190, in make_thunk
keep_lock=keep_lock)
File "/home/animation/myLib/Theano/theano/gof/cc.py", line 1131, in compile
keep_lock=keep_lock)
File "/home/animation/myLib/Theano/theano/gof/cc.py", line 1589, in cthunk_factory
key=key, lnk=self, keep_lock=keep_lock)
File "/home/animation/myLib/Theano/theano/gof/cmodule.py", line 1114, in module_from_key
module = self._get_from_key(key)
File "/home/animation/myLib/Theano/theano/gof/cmodule.py", line 1013, in _get_from_key
return self._get_module(name)
File "/home/animation/myLib/Theano/theano/gof/cmodule.py", line 687, in _get_module
self.module_from_name[name] = dlimport(name)
File "/home/animation/myLib/Theano/theano/gof/cmodule.py", line 302, in dlimport
rval = import(module_name, {}, {}, [module_name])
RuntimeError: ('The following error happened while compiling the node', GpuDnnConv{algo='small', inplace=True}(GpuContiguous.0, GpuContiguous.0, GpuAllocEmpty.0, GpuDnnConvDesc{border_mode='valid', subsample=(1, 1), conv_mode='conv', precision='float32'}.0, Constant{1.0}, Constant{0.0}), '\n', 'could not create cuDNN handle: CUDNN_STATUS_INTERNAL_ERROR')
Process finished with exit code 1
can anyone help me with this? Thanks a lots.
The text was updated successfully, but these errors were encountered: