occur "cudamat.cudamat.CUDAMatException: CUBLAS error." when running multimodal_dbm example

[Demoscai]

hi nitish srivastava
@nitishsrivastava
I have problems when running multimodal_dbm example, like this:
Train Step: 0Traceback (most recent call last):
File "/home/meitu299/deepnet/deepnet/trainer.py", line 60, in
main()
File "/home/meitu299/deepnet/deepnet/trainer.py", line 54, in main
model.Train()
File "/home/meitu299/deepnet/deepnet/neuralnet.py", line 631, in Train
self.GetTrainBatch()
File "/home/meitu299/deepnet/deepnet/neuralnet.py", line 524, in GetTrainBatch
self.GetBatch(self.train_data_handler)
File "/home/meitu299/deepnet/deepnet/dbm.py", line 264, in GetBatch
super(DBM, self).GetBatch(handler=handler)
File "/home/meitu299/deepnet/deepnet/neuralnet.py", line 512, in GetBatch
data_list = handler.Get()
File "/home/meitu299/deepnet/deepnet/datahandler.py", line 627, in Get
batch = self.gpu_cache.Get(self.batchsize, get_last_piece=self.get_last_piece)
File "/home/meitu299/deepnet/deepnet/datahandler.py", line 396, in Get
self.LoadData()
File "/home/meitu299/deepnet/deepnet/datahandler.py", line 327, in LoadData
self.data[i] = cm.CUDAMatrix(mat)
File "/home/meitu299/deepnet/cudamat/cudamat.py", line 195, in init
raise generate_exception(err_code)
cudamat.cudamat.CUDAMatException: CUBLAS error.

and the RAM is 8G and gpu memory is 3G in my computer ,CUDA6.0
I follow your INSTALL, but always happen this
could tell how to resolve this ? It's a bug?
thx

[cbalint13]

Try reduce batch size from 128 to 100.

[Demoscai]

I have try to reduce batch size to 50, but it doesn't work

[jormansa]

Try to fix the value "gpu_memory" of your .pbtxt file to "2G" or "2.5G"

[Demoscai]

thanks , that's OK

[tengshaofeng]

thanks to you in advanved
i have the similar problem.
when i run the example of ff,i set the steps from 1000000 to 10000, the batchsize from 100 to 10,the
gpu_memory from 2G to 0.1G,the main_memmory from 4G to 0.7G.
but when i come to the setp 499, it still comes to the problem like this:

File "/home/tbq/Downloads/deepnet-master/deepnet/softmax_layer.py", line 65, in GetLoss
perf.correct_preds = temp.sum()
File "/home/tbq/Downloads/deepnet-master/cudamat/cudamat.py", line720, in sum
return vdot(self,CUDAMatrix.ones.slice(0,self.shape[0]*self.shape[1]))
File "/home/tbq/Downloads/deepnet-master/cudamat/cudamat.py", line1650 in vdot
raise generate_exception(err_code.value)
cudamat.cudamat.CUDAMatException: CUBLAS error.

and the RAM is 1G and gpu memory is 256M in my computer ,CUDA5.5

when i try the dbm and rbm ,it is also comes to the problem
i want to know whether my cpu and gpu is not satisfy the demand.
thx

[tengshaofeng]

sorry,english is not mother tongue. in addition,gcc:4.6.3

[jnhwkim]

In my case, I decreased gpu_mem as 1G in run_all_dbn.sh though my gpu memory is 4G (NVIDIA GeForce GTX 780M 4096 MB).

[chaojiewang94]

thank you all , ruducing the gpu_mem really helps , and the code strat to work , but at the end of trainning the first layer , the bug happens again , is the gpu_mem still too large?
and what will happen if i reduce the gpu_mem

Thank you all a lot if anyone can help me