Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The GPU failed to allocate memory when started to run epoch 2 with fcnTrain.m #36

Closed
daofeng2007 opened this issue Aug 13, 2016 · 2 comments

Comments

@daofeng2007
Copy link

daofeng2007 commented Aug 13, 2016

I used Pascal Titan X to run fcnTrain.m. The first epoch went smoothly with 5.5Hz of GPU. However, when it finished epoch 01 and started to do epoch 02, following error showed up. I inspected size of ims. This variable size is 798 kb, and my GPU has 12GB.
Matconvnet was compiled under :Windows 10+Pascal Titan X +CUDN 8.0RC+CuDNN 5.1

train: epoch 02: 1/ 56:Error using gpuArray
The GPU failed to allocate memory. To continue, reset the GPU by running 'gpuDevice(1)'. If this problem persists,
partition your computations into smaller pieces.

Error in getBatch (line 91)
ims = gpuArray(ims) ;

Error in fcnTrain>@(imdb,batch)getBatch(imdb,batch,opts,'prefetch',nargout==0) (line 108)
fn = @(imdb,batch) getBatch(imdb,batch,opts,'prefetch',nargout==0) ;

Error in cnn_train_dag>process_epoch (line 197)
inputs = state.getBatch(state.imdb, batch) ;

Error in cnn_train_dag (line 83)
[stats.train(epoch),prof] = process_epoch(net, state, opts, 'train') ;

Error in fcnTrain (line 99)
info = cnn_train_dag(net, imdb, getBatchWrapper(bopts), ...

Following is the epoch 01

capture

@daofeng2007
Copy link
Author

I found the solution to this error. This error is actually not specific to fcnTrain.m. It happened randomly when I run other matconvnet examples. It turned out if I compiled matconvnet under cuda 7.5, there's no such error showing up.

@hushunda
Copy link

兄弟,我也遇到了,你解决了吗,能留个联系方式吗

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants