Error using Caffe_, glog check error, please check log and clear mex

[Anylee2142]

As solutions of issues told,

i uncomment 31 line of script_faster_rcnn_demo.m,

replace 7.5 to 6.5 of Cuda driver

and finally i could stop MATLAB crash

but when run script_faster_rcnn_demo,

---

fast_rcnn startup done
GPU 1: free memory 939069440
Use GPU 1
Error using caffe_
glog check error, please check log and clear mex

Error in caffe.Net/forward_prefilled (line 109)
caffe_('net_forward', self.hNet_self);

Error in caffe.Net/forward (line 142)
self.forward_prefilled();

Error in proposal_im_detect (line 23)
output_blobs = caffe_net.forward(net_inputs);

Error in script_faster_rcnn_demo (line 54)
[boxes, scores] =
proposal_im_detect(proposal_detection_model.conf_proposal, rpn_net, im);

---

this message and caffe_log.2680 file.

what should i do to run??

tell me if any necessary further information needed

(Windows 7 , VS 2010 , 6.5 Cuda driver, MATLAB 2013 and 2015)

thank you for reading

[Anylee2142]

i've checked log file by executing script_faster_rcnn_demo

and i figure that the last two line of log file says

F0106 19:55:17.793050 7656 syncedmem.cpp:51] Check failed: error == cudaSuccess (2 vs. 0) out of memory
F0106 19:55:17.793050 7656 syncedmem.cpp:51] Check failed: error == cudaSuccess (2 vs. 0) out of memory

it says 'out of memory'.

is it because of my gpu? My gpu is gtx 650 ti.

if gpu is under Titan or tesla (ex. T20 ,T40 etc..), does it end before start?

i mean my running ends without this part below.
001763.jpg (500x375): time 0.201s (resize+conv+proposal: 0.150s, nms+regionwise: 0.052s)
004545.jpg (500x375): time 0.201s (resize+conv+proposal: 0.151s, nms+regionwise: 0.050s)
000542.jpg (500x375): time 0.192s (resize+conv+proposal: 0.151s, nms+regionwise: 0.041s)

[Cogito2012]

I have similar problem as yours, anyone had solved this problem?
///////////////////////////////////////////////////////////////////////////
Error using caffe_
glog check error, please check log and clear mex

Error in caffe.Net/reshape (line 171)
caffe_('net_reshape', self.hNet_self);

Error in caffe.Net/reshape_as_input (line 186)
self.reshape();

Error in fast_rcnn_train (line 112)
caffe_solver.net.reshape_as_input(net_inputs);

Error in Faster_RCNN_Train.do_fast_rcnn_train (line 7)
model_stage.output_model_file = fast_rcnn_train(conf, dataset.imdb_train, dataset.roidb_train, ...

Error in script_faster_rcnn_pku (line 54)
model.stage1_fast_rcnn = Faster_RCNN_Train.do_fast_rcnn_train(conf_fast_rcnn, dataset, model.stage1_fast_rcnn, opts.do_val);

[Anylee2142]

@Cogito2012
may i ask what your gpu is?

mine is gtx 650 ti

[Cogito2012]

@Anylee2142 My GPU is Quadro K620, its only 2G memory

[Anylee2142]

@Cogito2012
did you run
experiments/script_faster_rcnn_demo?
i thought my problem that i cannot run script_faster_rcnn_demo was because of my gpu.

but i googled and figured out that the gap between K620 and 650 ti is not that huge.

what's wrong with me..

can you tell me your computer specification?

[Cogito2012]

My computer is a workstation T1700
My errors happened when I run training file experiments/script_faster_rcnn_ZF0712.m,
And I can successfully run the test file experiments/script_faster_rcnn_demo.m, you can try to modify the configs in /script_faster_rcnn_demo.m:
opts.use_gpu = false;
or use the ZF net:
% model_dir = fullfile(pwd, 'output', 'faster_rcnn_final', 'faster_rcnn_VOC0712_vgg_16layers'); %% VGG-16
model_dir = fullfile(pwd, 'output', 'faster_rcnn_final', 'faster_rcnn_VOC0712_ZF'); %% ZF

[Anylee2142]

thank you for your advice. i can run demo now !

and if i may, why ' opts.use_gpu = false; ' make it run?

i mean obviously faster r-cnn code needs gpu to be runned.

Doesn't ' opts.use_gpu = false; ' this mean not using gpu?

Do you know difference between opts.use_gpu = true and opts.use_gpu = false ?

[Cogito2012]

I can also run demo with opts.use_gpu = true only for ZF net.
you can read the README.MD:

Requirements: hardware

GPU: Titan, Titan Black, Titan X, K20, K40, K80.

1. Region Proposal Network (RPN)
   • 2GB GPU memory for ZF net
   • 5GB GPU memory for VGG-16 net
2. Ojbect Detection Network (Fast R-CNN)
   • 3GB GPU memory for ZF net
   • 8GB GPU memory for VGG-16 net

but I think these requirements are only for training proccess, in testing process (script_faster_rcnn_demo.m), if you set opts.use_gpu = false, the /functions/nms/nms.m file will not use gpu, thus affects boxes_filter's speed.
As for proposal_im_detect、fast_rcnn_conv_feat_detect、fast_rcnn_im_detect, I didn't find anywhere to use gpu, maybe by caffe.set_mode_gpu().

[Cogito2012]

@Anylee2142 You should open /experiments/+Dataset/private/voc2007_devkit.m and voc2012_devkit.m, and set your own VOC dataset devkit directory.
Then run script_faster_rcnn_VOC0712_ZF or script_faster_rcnn_VOC2007_ZF,
If the same errors still appear, please set break point at the line (line 46, imdb_from_voc.m ):
imdb.image_ids = textread(sprintf(VOCopts.imgsetpath, image_set), '%s');
to see VOCopts.imgsetpath and check your VOC devkit dirs

[Anylee2142]

@Cogito2012
Before checking your reply,

i had simply renamed the folder i downloaded as what path indicate:= VOCdevkit2007

and put VOC2007 folder into VOCdevkit2007 because imdb.img_dir is
D:/MATLAB2013/faster_rcnn-master/datasets/VOCdevkit2007/VOC2007/JPEGImages

after doing this, it seemed to run but

while MATLAB console displayed some working message, same error appeared, after this,
i'm still on the same point. (error running train, it doesn't even start)

VOCopts.imgsetpath's dir is
'D:/MATLAB2013/faster_rcnn-master/datasets/VOCdevkit2007/VOC2007/ImageSets/Main/%s.txt'
many files are in Main folder.. i don't know what's wrong

this is information about VOCopts
PATH.zip

[Cogito2012]

@Anylee2142 I don't actually know whether your local VOC dataset directory is right, I put my VOC2007 at disk E:, because the path in matlab is not permitted too long.
And I suggest that not rename the VOCdevkit, such as
D:/MATLAB2013/faster_rcnn-master/datasets/VOCdevkit/VOC2007/JPEGImages

[Anylee2142]

@Cogito2012
hmm.. i got folder's name back as VOCdevkit but nothing changed.

May i ask that you tell me your local configuration of datasets, VOC2007 and VOCdevkit?

maybe following your configuration works for me.

[Cogito2012]

@Anylee2142
VOC2007 datasets in my disk E:/PASCAL VOC2007/VOCdevkit/

|-- VOCdevkit/
    |-- local/
    |-- results/
    |-- VOC2007/
        |-- Annotations/
            *.xml
        |-- ImageSets/
            |-- Action/
            |-- Layout/
            |-- Main/
                *.txt (84 text files:train+val+trainval+test)
            |-- Segmentation/
        |-- JPEGImages/
            *.jpg
        |-- SegmentationClass/
        |-- SegmentationObject/
    |-- VOCcode/
        *.m
    *.m

And in my /experiments/+Dataset/private/voc2007_devkit.m file:
path = 'E:/PASCAL VOC2007/VOCdevkit';

[Anylee2142]

@Cogito2012
I'm grateful for your good.

My VOCdevkit/VOC2007/ImageSets/Main has only 63 txt files.

they have name like train+val+trainval+test but lesser than 84.

Is the reason why MATLAB displays Error using textread because of missing files?

If i may ask, Could you upload your Imagesets zip folder please?

[Cogito2012]

@Anylee2142
The 84 files are named as:

class_train.txt (20 files)
train.txt
class_val.txt (20 files)
val.txt
class_trainval.txt (20 files)
trainval.txt
class_test.txt (20 files)
test.txt

In the faster-rcnn training, only the trainval.txt will be used，if you didn't download test datasets, there will be 63 files, but will not affect the training.

Here is my ImageSets

[Anylee2142]

@Cogito2012
i really don't know what's wrong with my configuration..

why MATLAB can't find trainval.txt .. it's obviously there.

anyway, my testing demo problem is solved, i'm closing this issue.

thank you Cogito2012

[1SoSo]

@Anylee2142 and @Cogito2012 : I could run script_faster_rcnn_VOC2007_ZF without any problem with the original VOC2007 Dataset. Now I use my own dataset and I changed all path variables to this new dataset which has exactly the same architecture as VOC2007. I cleared all cachefiles and I get exactly this error.

Only differences: I am more interested in the RPN and hence I have only 2 object classes, no poses and no parts. Do you know if this error can occur if you have wrong prototxt files or so? I had already a problem to create the roidb, where I set my source path manually.