Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Training of py-faster-rcnn on ImageNet #9

Closed
ghost opened this issue Nov 1, 2015 · 66 comments
Closed

Training of py-faster-rcnn on ImageNet #9

ghost opened this issue Nov 1, 2015 · 66 comments

Comments

@ghost
Copy link

ghost commented Nov 1, 2015

I revised the codes in lib/datasets to train on the ImageNet detection data: I create an ilsvrc.py (like the pascal_voc.py) to account for the ImageNet data and modify the corresponding codes in factory.py. Then I run the experiment script ./experiments/scripts/faster_rcnn_alt_opt.sh and everything seems to be correct at first, but then the training seems to get stuck with the following information never moving forward again...

Loading pretrained model weights from data/imagenet_models/ZF.v2.caffemodel
Solving...
I1101 15:18:12.302584 40312 solver.cpp:242] Iteration 0, loss = 1.10525
I1101 15:18:12.302647 40312 solver.cpp:258]     Train net output #0: rpn_cls_loss = 0.785905 (* 1 =     0.785905 loss)
I1101 15:18:12.302659 40312 solver.cpp:258]     Train net output #1: rpn_loss_bbox = 0.319344 (* 1 = 0.319344 loss)
I1101 15:18:12.302670 40312 solver.cpp:571] Iteration 0, lr = 0.001

Could anyone give me some suggestions? Thanks!
@GregorySenay
Copy link

Same thing for me. I have reduced the pascal voc dataset to couple of classes.
The data loading is okay, number of classes changed, but it is stacks like you on the Iteration 0.

I suspect some errors in the roidb.

@sunshineatnoon
Copy link

@Jianchao-ICT I am also interested in training a model on ImageNet, so all I need to do is to change code in lib/datasets/ilsvrc.py and factory.py ? Do I need to change --imdb voc_2007_trainval \ in ./experiments/scripts/faster_rcnn_alt_opt.sh ?

@ghost
Copy link
Author

ghost commented Nov 3, 2015

@SenayG Hi, have you tried to run it without changing any code?
Well, now I located the problem by printing some information manually in the SolverWrapper.train_model in ./lib/fast_rcnn/train.py and the training gets stuck at the fourth call of self.solver.step(1). Since it will not print out the information in each iteration, so we only stop at Iteration 0.
Anyway, I am still painstakingly trying to fix it...

@ghost
Copy link
Author

ghost commented Nov 3, 2015

@sunshineatnoon Yes, you also need to change the .sh training script to use your dataset. You may also specify the dataset in the command line using --imdb YOUR_DATASET.

@sunshineatnoon
Copy link

@Jianchao-ICT Thx. I have trained a fast rcnn model on ImageNet, it seems the method to train faster rcnn is the same, right?

@GregorySenay
Copy link

@Jianchao-ICT as I suspected for myself it was due to the data (maybe the same thing for you).
Look the py-faster-rcnn/lib/roi_data_layer/minibatch.py file and line 80 you can print the value for each batch with "blobs"

print blobs

and look the last values before your crash, if some values are empty the crash is normal, you have to check again you dataset.

@ghost
Copy link
Author

ghost commented Nov 3, 2015

@sunshineatnoon Well, I guess training the two is basically the same since the core of faster r-cnn is also implemented using fast rcnn. However, I still came across the above problem...

@ghost
Copy link
Author

ghost commented Nov 3, 2015

@SenayG Oh, GREAT THANKS! I will try it now.

@ghost
Copy link
Author

ghost commented Nov 3, 2015

@SenayG Hi, I have added print blobs to the line you mention. And the printing information is as follows (sorry that they are so long). But I do not see something like empty values. Could you help me check for it? Thanks!

Loading pretrained model weights from data/imagenet_models/ZF.v2.caffemodel
Solving...
{'gt_boxes': array([[ 411.20001221,  254.3999939 ,  534.40002441,  451.20001221,   97.        ],
   [ 136.        ,  246.3999939 ,  280.        ,  465.6000061 ,   97.        ],
   [ 248.        ,  240.        ,  390.3999939 ,  448.        ,   97.        ],
   [ 481.6000061 ,  283.20001221,  638.40002441,  456.        ,   97.        ]], dtype=float32), 'data': array([[[[  4.01989985,   4.45739985,   4.95739985, ...,  -9.16759968,
      -16.29260063, -21.98010063],
     [  5.76990032,   5.82458735,   5.88708735, ...,  -8.75744343,
      -15.11681938, -20.23010063],
     [  7.95740032,   7.4925561 ,   6.9534936 , ...,  -8.08947468,
      -13.55041313, -17.98010063],
     ..., 
     [-50.04259872, -48.97619247, -46.67150497, ..., -38.23009872,
      -33.35119247, -27.85510063],
     [-34.35509872, -37.41759872, -40.98009872, ..., -23.77697372,
      -15.47228622,  -7.98010015],
     [-22.98010063, -29.10509872, -37.04259872, ..., -11.41760063,
       -1.85509968,   6.01989985]],

    [[-22.94650078, -22.50900078, -21.82150078, ..., -43.82149887,
      -43.38399887, -42.94649887],
     [-22.07150078, -22.01681328, -21.84884453, ..., -42.94649887,
      -41.77071762, -40.75899887],
     [-21.00900078, -21.41915703, -21.88790703, ..., -41.85665512,
      -39.79415512, -38.07149887],
     ..., 
     [-46.00899887, -41.44259262, -35.15353012, ..., -47.54806137,
      -30.46603012, -15.94649982],
     [-44.13399887, -37.21603012, -28.86056137, ..., -19.04806137,
       -8.69259262,  -0.57149982],
     [-41.94649887, -34.07149887, -25.19650078, ...,   2.99100018,
        6.86600018,   9.05350018]],

    [[-25.77169991, -25.33419991, -24.77169991, ..., -45.77170181,
      -44.08420181, -42.77170181],
     [-26.20919991, -26.15451241, -26.05685616, ..., -45.85373306,
      -45.31467056, -44.95920181],
     [-26.45919991, -26.89669991, -27.40841866, ..., -45.58810806,
      -46.83810806, -47.95920181],
     ..., 
     [-51.08420181, -36.78342056, -18.31076241, ..., -28.17013931,
      -22.21701431, -15.27169991],
     [-32.64670181, -20.28732681,  -5.38888741, ...,  -6.17013836,
        7.09548569,  18.85329819],
     [-17.77169991,  -7.70919991,   3.41580009, ...,  12.04080009,
       29.79079819,  44.22829819]]]], dtype=float32), 'im_info': array([[ 600.        ,  800.        ,    1.60000002]], dtype=float32)}
I1103 14:39:37.399827 37785 solver.cpp:242] Iteration 0, loss = 1.10525
I1103 14:39:37.400996 37785 solver.cpp:258]     Train net output #0: rpn_cls_loss = 0.785905 (*    1 = 0.785905 loss)
I1103 14:39:37.401373 37785 solver.cpp:258]     Train net output #1: rpn_loss_bbox = 0.319344 (* 1 = 0.319344 loss)
I1103 14:39:37.401665 37785 solver.cpp:571] Iteration 0, lr = 0.001
{'gt_boxes': array([[ 317.64706421,   65.77539825,  625.66845703,  596.79144287,  178.        ],
   [ 630.48126221,  142.78074646,  802.13903809,  595.18719482,  178.        ],
   [   1.60427809,   16.04277992,  449.19787598,  596.79144287,  182.        ]], dtype=float32), 'data': array([[[[  1.39019897e+02,   1.39019897e+02,   1.39019897e+02, ...,
        7.11499634e+01,   7.44115601e+01,   7.70198975e+01],
     [  1.38584900e+02,   1.38584900e+02,   1.38584900e+02, ...,
        7.17175980e+01,   7.21227417e+01,   7.22348938e+01],
     [  1.38078232e+02,   1.38078232e+02,   1.38078232e+02, ...,
        7.25131989e+01,   6.92919464e+01,   6.62532272e+01],
     ..., 
     [  9.64950256e+01,   8.99192886e+01,   8.04756699e+01, ...,
       -6.37412262e+01,  -5.87805023e+01,  -5.58634605e+01],
     [  9.13698730e+01,   9.24639282e+01,   9.25231552e+01, ...,
       -6.10821762e+01,  -5.89387131e+01,  -5.99351120e+01],
     [  8.70198975e+01,   9.39798965e+01,   1.01328232e+02, ...,
       -5.86048241e+01,  -5.88067589e+01,  -6.29800949e+01]],

    [[  1.38053497e+02,   1.38053497e+02,   1.38053497e+02, ...,
        9.13434753e+01,   9.00968323e+01,   8.90534973e+01],
     [  1.37618500e+02,   1.37618500e+02,   1.37618500e+02, ...,
        9.20873337e+01,   9.09668274e+01,   8.99235001e+01],
     [  1.37111832e+02,   1.37111832e+02,   1.37111832e+02, ...,
        9.31118317e+01,   9.21272659e+01,   9.10534973e+01],
     ..., 
     [  1.23878540e+02,   1.20808174e+02,   1.14971565e+02, ...,
       -4.11328888e+01,  -2.67664261e+01,  -1.12380953e+01],
     [  1.26878510e+02,   1.24570831e+02,   1.20317894e+02, ...,
       -3.77918243e+01,  -2.69656506e+01,  -1.60315208e+01],
     [  1.29053497e+02,   1.27313492e+02,   1.24178505e+02, ...,
       -3.52514000e+01,  -2.72498436e+01,  -1.99465008e+01]],

    [[  1.31228302e+02,   1.31228302e+02,   1.31228302e+02, ...,
        1.06938324e+02,   1.07706635e+02,   1.08228302e+02],
     [  1.30793304e+02,   1.30793304e+02,   1.30793304e+02, ...,
        1.07934464e+02,   1.08160484e+02,   1.08228302e+02],
     [  1.30286636e+02,   1.30286636e+02,   1.30286636e+02, ...,
        1.09353424e+02,   1.08810829e+02,   1.08228302e+02],
     ..., 
     [  1.31645111e+02,   1.29676804e+02,   1.25583656e+02, ...,
       -9.91133976e+00,   1.87424850e+00,   1.68200645e+01],
     [  1.29403290e+02,   1.29875275e+02,   1.29425476e+02, ...,
       -7.24253225e+00,   1.09261334e-01,   8.75326347e+00],
     [  1.27228302e+02,   1.29403290e+02,   1.31469971e+02, ...,
       -5.20666695e+00,  -1.42337227e+00,   2.22830009e+00]]]], dtype=float32), 'im_info': array([[ 600.        ,  802.        ,    1.60427809]], dtype=float32)}
{'gt_boxes': array([[ 262.3999939 ,   44.79999924,  712.        ,  480.        ,  182.        ],
   [ 113.59999847,  379.20001221,  350.3999939 ,  596.79998779,  199.        ]], dtype=float32), 'data': array([[[[ -96.98010254,  -96.98010254,  -96.91760254, ...,  -97.98010254,
       -97.98010254,  -97.98010254],
     [ -96.54260254,  -96.54260254,  -96.50744629, ...,  -97.98010254,
       -97.98010254,  -97.98010254],
     [ -95.91760254,  -95.91760254,  -95.91760254, ...,  -97.98010254,
       -97.98010254,  -97.98010254],
     ..., 
     [  -8.35509968,   -8.73791218,   -9.10509968, ...,    3.07849383,
         1.83240008,    0.95739996],
     [  -6.29260015,   -7.22228718,   -8.25744343, ...,    4.83240032,
         3.89880633,    3.26990008],
     [  -4.98010015,   -6.29260015,   -7.79260015, ...,    6.14489985,
         5.45739985,    5.01989985]],

    [[-100.94650269, -100.94650269, -100.88400269, ..., -105.94650269,
      -105.94650269, -105.94650269],
     [-100.50900269, -100.50900269, -100.47384644, ..., -105.94650269,
      -105.94650269, -105.94650269],
     [ -99.88400269,  -99.88400269,  -99.88400269, ..., -105.94650269,
      -105.94650269, -105.94650269],
     ..., 
     [ -17.32150078,  -17.70431328,  -18.07150078, ...,   -9.88790607,
       -11.13399982,  -12.00899982],
     [ -15.25900078,  -16.18868828,  -17.22384453, ...,   -8.13399982,
        -9.06759357,   -9.69649982],
     [ -13.94650078,  -15.25900078,  -16.75900078, ...,   -6.82149982,
        -7.50899982,   -7.94649982]],

    [[ -97.771698  ,  -97.771698  ,  -97.709198  , ..., -113.771698  ,
      -113.771698  , -113.771698  ],
     [ -97.334198  ,  -97.334198  ,  -97.29904175, ..., -113.771698  ,
      -113.771698  , -113.771698  ],
     [ -96.709198  ,  -96.709198  ,  -96.709198  , ..., -113.771698  ,
      -113.771698  , -113.771698  ],
     ..., 
     [ -23.14669991,  -23.52951241,  -23.89669991, ...,  -16.71310616,
       -17.95919991,  -18.83419991],
     [ -21.08419991,  -22.01388741,  -23.04904366, ...,  -14.95919991,
       -15.89279366,  -16.52169991],
     [ -19.77169991,  -21.08419991,  -22.58419991, ...,  -13.64669991,
       -14.33419991,  -14.77169991]]]], dtype=float32), 'im_info': array([[ 600.        ,  800.        ,    1.60000002]], dtype=float32)}
{'gt_boxes': array([], shape=(0, 5), dtype=float32), 'data': array([[[[ 142.01989746,  142.01989746,  142.36988831, ...,  141.77655029,
       143.63317871,  146.01989746],
     [ 141.54989624,  141.32897949,  141.37506104, ...,  141.06529236,
       142.50361633,  144.6098938 ],
     [ 141.13656616,  140.61172485,  140.28045654, ...,  140.43359375,
       141.2769165 ,  143.13656616],
     ..., 
     [ 141.01989746,  141.01989746,  141.01989746, ...,  141.77655029,
       141.61657715,  141.01989746],
     [ 141.01989746,  141.01989746,  141.01989746, ...,  141.77655029,
       141.61657715,  141.01989746],
     [ 141.01989746,  141.01989746,  141.01989746, ...,  141.77655029,
       141.61657715,  141.01989746]],

    [[ 133.05349731,  131.64349365,  130.40348816, ...,  130.05349731,
       130.86013794,  132.05349731],
     [ 132.58349609,  131.39439392,  130.40348816, ...,  129.92662048,
       130.39013672,  131.58349609],
     [ 132.05349731,  131.1683197 ,  130.57460022, ...,  130.15882874,
       130.09347534,  131.28683472],
     ..., 
     [ 131.05349731,  131.05349731,  131.05349731, ...,  131.81015015,
       131.650177  ,  131.05349731],
     [ 131.05349731,  131.05349731,  131.05349731, ...,  131.81015015,
       131.650177  ,  131.05349731],
     [ 131.05349731,  131.05349731,  131.05349731, ...,  131.81015015,
       131.650177  ,  131.05349731]],

    [[ 115.228302  ,  114.28829956,  113.69496918, ...,  114.95834351,
       114.63162231,  115.228302  ],
     [ 114.75830078,  114.26010132,  114.32946777, ...,  116.01271057,
       115.19249725,  115.228302  ],
     [ 114.46163177,  114.57130432,  115.55246735, ...,  117.8072052 ,
       116.26908875,  115.81163025],
     ..., 
     [ 115.228302  ,  115.228302  ,  115.228302  , ...,  115.98495483,
       115.82498169,  115.228302  ],
     [ 115.228302  ,  115.228302  ,  115.228302  , ...,  115.98495483,
       115.82498169,  115.228302  ],
     [ 115.228302  ,  115.228302  ,  115.228302  , ...,  115.98495483,
       115.82498169,  115.228302  ]]]], dtype=float32), 'im_info': array([[ 600.        ,  773.        ,    1.54639173]], dtype=float32)}

@ghost
Copy link
Author

ghost commented Nov 3, 2015

@SenayG Oh, I see, the last gt_boxes is an empty []. I will check it again. Thanks for your really useful hints!

@GregorySenay
Copy link

So clearly the issue is here : {'gt_boxes': array([], shape=(0, 5), dtype=float32),

@GregorySenay
Copy link

Your welcome! Good to help you to debug!

@ghost
Copy link
Author

ghost commented Nov 3, 2015

@SenayG Thanks! You are really nice 👍

@ghost
Copy link
Author

ghost commented Nov 3, 2015

@SenayG Oh, the problem is found. It is due to the image ILSVRC2013_val_00003481.JPEG in the fourth batch. This image strangely has no ground truth annotations and causes gt_boxes to be empty...
Oh, damn it, how could such images still appear in the dataset...

@GregorySenay
Copy link

Good the issue, can now be close, I think

@ghost
Copy link
Author

ghost commented Nov 4, 2015

@SenayG Hi, I have closed it.

@ghost ghost closed this as completed Nov 4, 2015
@sunshineatnoon
Copy link

@Jianchao-ICT I am training the model on imagenet, but I have few problems:

  1. do I need to change files in /py-faster-rcnn/models/ZF/faster_rcnn_alt_opt/ to specify my own classes?
  2. I've encounter a problem:
File "/py-faster-rcnn/tools/../lib/datasets/imdb.py", line 108, in append_flipped_images
    assert (boxes[:, 2] >= boxes[:, 0]).all()
AssertionError

The command I used to train is:

./tools/train_faster_rcnn_alt_opt.py --gpu 0 --net_name ZF --weights data/imagenet_models/ZF.v2.caffemodel --imdb imagenet_train --cfg experiments/cfgs/faster_rcnn_alt_opt.yml

Did you have the same problem?

@ghost
Copy link
Author

ghost commented Nov 5, 2015

@sunshineatnoon

  1. Well, I am not sure whether this method will work. To account for the classes of ImageNet, I add a file ilsvrc.py inside ./lib/datasets like the original pascal_voc.py.

  2. Haha, I have met exactly the same problem. This is due to the Pascal Voc's annotations are 1-based while the ImageNet's annotations are 0-based in xmin, xmax, ymin, ymax. So you just need to change the code in the function def _load_pascal_annotation(self, index) in your added ilsvrc.py. Specifically, the following four lines need to get the - 1 removed.

    x1 = float(get_data_from_tag(obj, 'xmin')) - 1
    y1 = float(get_data_from_tag(obj, 'ymin')) - 1
    x2 = float(get_data_from_tag(obj, 'xmax')) - 1
    y2 = float(get_data_from_tag(obj, 'ymax')) - 1

@sunshineatnoon
Copy link

@Jianchao-ICT I've got rid of the -1 operation, but I still got the same error. Then I found this guy: n02958343_19208.xml had wrong annotations. There is only one object in the image, but there are four objects annotated in the .xml file. Changing the annotation file fixed my problem.

@ghost
Copy link
Author

ghost commented Nov 5, 2015

@sunshineatnoon Oh, driven crazy by this dataset: it has involved many gray images; some of the images have 0 ground truth object bounding boxes. Both these two have incurred some mistakes to my program. In fact I use the validation set for training since I have still been not very clear with the labels of the training set: there are more than 200 categories labeled in it.

Now I also see your problem. I just want to say this dataset seems not that user-friendly...

@ghost
Copy link
Author

ghost commented Nov 9, 2015

@sunshineatnoon Hi, how is your training now? I now come across the following error in the Stage 1 Fast RCNN training. Have you got similar things?

Traceback (most recent call last):
  File "/home/code/lijianchao/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/code/lijianchao/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "./tools/train_faster_rcnn_alt_opt.py", line 196, in train_fast_rcnn
    max_iters=max_iters)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/fast_rcnn/train.py", line 138, in train_net
    model_paths = sw.train_model(max_iters)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/fast_rcnn/train.py", line 104, in train_model
    self.solver.step(1)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/roi_data_layer/layer.py", line 143, in forward
    blobs = self._get_next_minibatch()
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/roi_data_layer/layer.py", line 62, in _get_next_minibatch
    return get_minibatch(minibatch_db, self._num_classes)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/roi_data_layer/minibatch.py", line 55, in get_minibatch
    num_classes)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/roi_data_layer/minibatch.py", line 125, in _sample_rois
    roidb['bbox_targets'][keep_inds, :], num_classes)
  File "/home/code/lijianchao/py-faster-rcnn/tools/../lib/roi_data_layer/minibatch.py", line 176, in _get_bbox_regression_labels
    bbox_targets[ind, start:end] = bbox_target_data[ind, 1:]
ValueError: could not broadcast input array from shape (4) into shape (0)

@sunshineatnoon
Copy link

@Jianchao-ICT No, I didn't encounter this problem, so it might caused by your dataset. I got other errors (#13) during Stage 1 Fast RCNN training. I fixed that error and still training now, hope no more errors...

@ghost
Copy link
Author

ghost commented Nov 9, 2015

@sunshineatnoon Oh, I train on the ImageNet detections. Well, I will try to figure it out... I also hope no more errors...

@ghost
Copy link
Author

ghost commented Nov 10, 2015

@sunshineatnoon Hi, I have figured out the reason... I made a too naive mistake: I forgot to change the num_output in .pt (prototxt) from 21 (PASCAL VOC) to 201 (ImageNet)...

@sunshineatnoon
Copy link

@Jianchao-ICT Good for you! You trained on the whole dataset of imagenet? I only trained on two classes on ImageNet and it took a long long time for training.

@ghost
Copy link
Author

ghost commented Nov 10, 2015

@sunshineatnoon Yeah, I train on the 201 category (with background as one category) but I use the validation set for training since I notice that the training set has some labels that are not in the 200 categories?

@ghost ghost reopened this Nov 10, 2015
@ghost ghost changed the title The training seems to get stuck Training of py-faster-rcnn on ImageNet Nov 10, 2015
@leefionglee
Copy link

Hi all:
I would like to train py-faster-rcnn on my own dataset, but what is exactly the dataformat? i.e. the images, annotations, train, val? Can anyone post an example annotation file here? As I found one post here: https://github.com/zeyuanxy/fast-rcnn/tree/master/help/train, in which the annotation is text, while I also found PASCAL is XML. What exactly it is? Any reference blogs or tutorials will be highly appreciated. Thanks

@ednarb29
Copy link

ednarb29 commented Jul 7, 2016 •
edited
Loading

@JohnnyY8

It should look more like this, if you refer to the original pascal_voc structure:

for year in ['2016']:
for split in ['train', 'test']:
name = 'vid_{}_{}'.format(year, split)
__sets[name] = (lambda split=split, year=year: vid(split, year))

Then you have to create the function, that creates the dataset with the call vid(split, year).

IMDB is actually the image database, where the data is stored for training, filled by your own function.

If you have a look into the shell script, you will note that the shell script just calls the python script. You can do both ways but the shell script might be more comfortable, even for testing later on.

Does this help a bit?

@JohnnyY8
Copy link

JohnnyY8 commented Jul 7, 2016

@ednarb29 Thanks for your time!
The explaination about shell script and python script is good for me. I am confused about two of them because too many parameters in shell script. And I do not know how to set the right path about train_imdb or test_imdb. As for lambda expression, I am a new fisher.
Thanks again! About the questions for training, I will reply on #243.

@ednarb29
Copy link

ednarb29 commented Jul 7, 2016

The shell script just defines the parameters and calls the python script for training and testing. That is only for automation, so later you just have to call the shell script and everything is done for you.

@JohnnyY8
Copy link

JohnnyY8 commented Jul 7, 2016

@ednarb29 So if I modify the codes and set training or test data path rightly, just change some parameters for shell script will be ok for training and testing, right ?

@ednarb29
Copy link

Yes, that all depends somehow on the other changes you have to do for the new dataset, but the idea is an easy use for later experiments when it is configured.

@JohnnyY8
Copy link

JohnnyY8 commented Aug 2, 2016

@ednarb29 @Jianchao-ICT Hi!
I want to train faster r-cnn on ImageNet DET dataset by following command:
sudo rm -f data/cache/* && sudo ./tools/train_net.py --gpu 0 --iters 100000 --imdb DET_train --weights data/imagenet_models/ZF.v2.caffemodel --cfg experiments/cfgs/faster_rcnn_end2end.yml --solver models/imagenet_det/ZF/faster_rcnn_end2end/solver.prototxt
But I got this error:
image
I have checked all xml files, if some of them do not have any object or the object's name beyond the 200 classes, I decide to ingore these xml files. What's more, I also remove -1 in mydataset.py or imdb.py, but it does not work.
Any suggestions? Thanks!

@leejiajun
Copy link

@JohnnyY8 it seems that caffemodel and prototxt do not match.

@JohnnyY8
Copy link

JohnnyY8 commented Aug 3, 2016

@leejiajun Yes, I solver it by changing
--solver models/imagenet_det/ZF/faster_rcnn_end2end/solver.prototxt to
--solver models/pascal_voc/ZF/faster_rcnn_end2end/solver.prototxt. That means it still load the old train.prototxt under pascal_voc, right?
But I got another error:
image
Do you know what happened?

@leejiajun
Copy link

leejiajun commented Aug 3, 2016 •
edited
Loading

@JohnnyY8 argmax is an empty sequence. this problem could be comes from too small or too large the size of image of your training set.

BTW: http://blog.csdn.net/jiajunlee/article/details/50470897

@JohnnyY8
Copy link

JohnnyY8 commented Aug 3, 2016

@leejiajun Thanks for your blog.
So that means I need to remove all too small or too large images in traning set by some script, right?
Is the ratio (width / height) range in 0.117-15.500 ?

@leejiajun
Copy link

@JohnnyY8
I removed.

@JohnnyY8
Copy link

JohnnyY8 commented Aug 4, 2016

@leejiajun Hi！
I have removed all too small or too large images. All remainings' ratio are in (0.18, 15.000).
But I meet another problem, the iteration just stopped when it started, like following:
I0804 11:28:13.044482 21960 solver.cpp:229] Iteration 0, loss = 6.54452
I0804 11:28:13.044561 21960 solver.cpp:245] Train net output #0: bbox_loss = 0.305254 (* 1 = 0.305254 loss)
I0804 11:28:13.044575 21960 solver.cpp:245] Train net output #1: cls_loss = 5.45421 (* 1 = 5.45421 loss)
I0804 11:28:13.044585 21960 solver.cpp:245] Train net output #2: rpn_cls_loss = 0.614111 (* 1 = 0.614111 loss)
I0804 11:28:13.044595 21960 solver.cpp:245] Train net output #3: rpn_loss_bbox = 0.379346 (* 1 = 0.379346 loss)
I0804 11:28:13.044607 21960 sgd_solver.cpp:106] Iteration 0, lr = 0.001
Is this as your blog said "play dead"?

@leejiajun
Copy link

@JohnnyY8 Probably! I remenber that people have discussed your problem in this issue.

@JohnnyY8
Copy link

JohnnyY8 commented Aug 5, 2016

@leejiajun Yes, but I have checked all xml files more than once, all remainings maybe ok.
I still try to print blobs information to find the problem. Do you have any other suggestion please?

@JohnnyY8
Copy link

JohnnyY8 commented Aug 5, 2016

@Jianchao-ICT @GregorySenay Hi!
I meet the same problem that the iteration get stuck when it started. And I try to print blobs information, there seems not empty variable. What's more, I try to check all remaining xml files are ok by python script.
Could you have any suggestion?
Thanks!

@GregorySenay
Copy link

Can you print the blobs during the iteration and paste the end of the log ?

@JohnnyY8
Copy link

JohnnyY8 commented Aug 5, 2016

@GregorySenay No problem！
But there is too much, the following is last part of it. Thanks a lot!

{'gt_boxes': array([[ 156.75675964,   82.88288116,  805.40539551,  598.19818115,   26.        ]], dtype=float32), 'data': array([[[[ -45.98009872,  -33.01259995,  -11.36759949, ...,  -98.62988281,
           -87.5300293 ,  -81.98010254],
         [ -30.68510056,  -22.25040817,   -8.17144394, ...,  -94.91837311,
           -84.74119568,  -79.65260315],
         [  -5.15510035,   -4.28644419,   -2.83650637, ...,  -88.7232132 ,
           -80.08614349,  -75.76760101],
         ..., 
         [  32.94458008,   37.6867485 ,   45.60224152, ...,   47.19591141,
            61.69502258,   68.94458008],
         [  47.37518311,   57.46916199,   74.31774902, ...,   57.00600433,
            74.58545685,   83.37518311],
         [  56.01990128,   69.31990051,   91.51989746, ...,   62.8827858 ,
            82.30752563,   92.01989746]],

        [[   9.05350018,   12.04599953,   17.04099846, ...,  -14.27138996,
            -8.7214632 ,   -5.94649982],
         [  15.03849983,   17.47821808,   21.55052948, ...,  -11.77848244,
            -6.78216076,   -4.28399992],
         [  25.0284996 ,   26.54553032,   29.07771873, ...,   -7.61738825,
            -3.5451293 ,   -1.50900006],
         ..., 
         [  52.76559448,   59.74810028,   71.40309143, ...,   63.76848984,
            72.34997559,   76.64071655],
         [  71.08135986,   78.06385803,   89.71885681, ...,   70.52735901,
            83.11328888,   89.40625   ],
         [  82.05349731,   89.03599548,  100.69099426, ...,   74.57629395,
            89.56109619,   97.05349731]],

        [[  21.228302  ,   25.88330078,   33.65329742, ...,  -40.92908096,
           -34.82416153,  -31.77169991],
         [  26.88080025,   30.31968307,   36.05976868, ...,  -38.49187088,
           -33.12509155,  -30.44170189],
         [  36.3157959 ,   37.72476959,   40.07658005, ...,  -34.4237442 ,
           -30.28904915,  -28.22169876],
         ..., 
         [  83.90322876,   89.26475525,   98.21405792, ...,   90.79845428,
            94.50986481,   96.36557007],
         [  87.23336792,   94.44034576,  106.47002411, ...,   94.41313171,
           102.74504852,  106.91101074],
         [  89.228302  ,   97.540802  ,  111.415802  , ...,   96.57852173,
           107.67837524,  113.228302  ]]]], dtype=float32), 'im_info': array([[ 600.       ,  901.       ,    1.8018018]], dtype=float32)}
{'gt_boxes': array([[ 140.11976624,   23.35329247,  810.17962646,  562.27545166,  101.        ]], dtype=float32), 'data': array([[[[ -64.98010254,  -64.64509583,  -64.08843231, ...,  -68.97509766,
           -68.98010254,  -68.98010254],
         [ -64.98010254,  -64.75732422,  -64.38713837, ...,  -67.3000946 ,
           -67.12029266,  -66.9701004 ],
         [ -64.98010254,  -64.94380951,  -64.88350677, ...,  -64.51676178,
           -64.02986145,  -63.63010025],
         ..., 
         [ -94.08843994,  -93.34585571,  -92.11190033, ...,  -97.09790039,
           -96.53677368,  -96.08843994],
         [ -94.64511108,  -93.52955627,  -91.67584229, ...,  -97.65179443,
           -97.09344482,  -96.64511108],
         [ -94.98010254,  -93.64009857,  -91.41343689, ...,  -97.98510742,
           -97.42843628,  -96.98010254]],

        [[ -50.94650269,  -52.95649719,  -56.29649734, ...,  -61.93648911,
           -61.94649887,  -61.94649887],
         [ -52.28649902,  -53.51091766,  -55.54553986, ...,  -61.93816376,
           -61.94649506,  -61.94649506],
         [ -54.51316452,  -54.4322052 ,  -54.2976799 , ...,  -61.94094849,
           -61.94649887,  -61.94649887],
         ..., 
         [ -99.05484009,  -98.04983521,  -96.37984467, ..., -106.06430817,
          -105.56293488, -105.16317749],
         [ -99.61151123,  -98.60650635,  -96.93650818, ..., -106.61819458,
          -106.42671204, -106.27651978],
         [ -99.94650269,  -98.9414978 ,  -97.27149963, ..., -106.95150757,
          -106.94650269, -106.94650269]],

        [[-100.771698  ,  -98.09169769,  -93.6383667 , ..., -103.77670288,
          -103.771698  , -103.771698  ],
         [ -98.42669678,  -96.86894226,  -94.28044891, ..., -104.77835083,
          -104.4070816 , -104.10669708],
         [ -94.53003693,  -94.83711243,  -95.34738922, ..., -106.44277954,
          -105.46289062, -104.66336823],
         ..., 
         [-116.8800354 , -116.54502869, -115.9883728 , ..., -117.88950348,
          -117.20883942, -116.6633606 ],
         [-117.43670654, -117.10169983, -116.54504395, ..., -118.44338989,
          -117.15131378, -116.10668945],
         [-117.771698  , -117.43669128, -116.8800354 , ..., -118.77670288,
          -117.11669922, -115.771698  ]]]], dtype=float32), 'im_info': array([[ 600.        ,  898.        ,    1.79640722]], dtype=float32)}
{'gt_boxes': array([[   3.18302393,  183.02388   ,  558.6206665 ,  792.57293701,  124.        ]], dtype=float32), 'data': array([[[[-102.98010254, -101.21009827,  -99.26343536, ...,   16.09515572,
            22.23242378,   20.01989746],
         [-102.98010254, -101.60171509, -100.02306366, ...,   16.40118408,
            20.6073494 ,   17.80739975],
         [-102.98010254, -102.15779877, -101.12177277, ...,   17.82502937,
            18.45528793,   14.38240051],
         ..., 
         [ -97.47503662,  -88.94907379,  -78.49136353, ...,  -81.52783203,
           -95.53408813, -102.7326355 ],
         [-101.24517822,  -98.55821228,  -94.92288971, ...,  -90.3013382 ,
           -99.48808289, -100.8475647 ],
         [-102.98010254, -102.98010254, -102.48426819, ...,  -94.33868408,
          -101.30761719,  -99.98010254]],

        [[-115.94650269, -115.06150055, -114.08816528, ...,  100.62063599,
           113.70852661,  111.05349731],
         [-115.94650269, -115.45311737, -114.84779358, ...,   98.64407349,
           113.91978455,  114.59349823],
         [-115.94650269, -115.94650269, -115.81487274, ...,   98.12947083,
           113.98478699,  117.84933472],
         ..., 
         [-110.44143677,  -94.97698212,  -76.4625473 , ...,  -85.0682373 ,
          -103.93998718, -112.61654663],
         [-114.21157837, -109.03489685, -102.7697525 , ...,  -97.48362732,
          -109.49004364, -110.1031189 ],
         [-115.94650269, -115.5039978 , -114.87567139, ..., -103.1968689 ,
          -112.04403687, -108.94650269]],

        [[-122.771698  , -121.88669586, -120.9133606 , ...,   96.51211548,
           123.82070923,  132.228302  ],
         [-122.771698  , -122.27831268, -121.67298889, ...,   96.01977539,
           119.43886566,  126.47579956],
         [-122.771698  , -122.771698  , -122.64006805, ...,   98.00818634,
           114.30695343,  118.51997375],
         ..., 
         [-117.26663208, -103.46268463,  -86.97528076, ...,  -93.52438354,
          -113.71115112, -122.60671997],
         [-121.03677368, -116.68645477, -111.44192505, ..., -103.60436249,
          -118.849617  , -121.3500061 ],
         [-122.771698  , -122.771698  , -122.7008667 , ..., -108.24291992,
          -121.21420288, -120.771698  ]]]], dtype=float32), 'im_info': array([[ 796.        ,  600.        ,    1.59151196]], dtype=float32)}
{'gt_boxes': array([[  36.79999924,   57.59999847,  600.        ,  593.59997559,  124.        ]], dtype=float32), 'data': array([[[[ 109.01989746,  106.83239746,  104.01989746, ...,  146.08239746,
           144.33239746,  143.01989746],
         [ 108.58239746,  106.77770996,  104.43005371, ...,  146.10974121,
           144.82458496,  143.89489746],
         [ 108.08239746,  106.74255371,  104.96130371, ...,  146.08239746,
           145.43005371,  145.01989746],
         ..., 
         [ -88.85510254,  -90.52307129,  -92.90588379, ...,   -8.31603718,
             1.14099383,    7.89490032],
         [ -90.66760254,  -92.17150879,  -94.31994629, ...,  -18.15978813,
            -9.99963188,   -3.79259992],
         [ -91.98010254,  -93.29260254,  -95.16760254, ...,  -24.66760063,
           -18.23010063,  -12.98010063]],

        [[  69.05349731,   66.86599731,   64.05349731, ...,  127.11599731,
           126.49099731,  126.05349731],
         [  69.49099731,   67.68630981,   65.33865356, ...,  127.96365356,
           127.36599731,  126.92849731],
         [  70.30349731,   68.93630981,   67.11209106, ...,  128.99880981,
           128.46365356,  128.05349731],
         ..., 
         [-106.94650269, -108.64181519, -111.05978394, ...,   13.15896893,
            20.02224922,   24.61599922],
         [-107.50900269, -109.01290894, -111.16134644, ...,    4.84646893,
            11.29568672,   16.05349922],
         [-107.94650269, -109.25900269, -111.13400269, ...,   -0.75899994,
             4.6784997 ,    9.05350018]],

        [[ -81.771698  ,  -83.959198  ,  -86.771698  , ...,  -56.70920181,
           -58.45920181,  -59.77170181],
         [ -82.209198  ,  -84.0138855 ,  -86.36154175, ...,  -56.27170181,
           -57.77560806,  -58.89670181],
         [ -82.709198  ,  -83.99435425,  -85.71310425, ...,  -55.76779556,
           -56.92404556,  -57.77170181],
         ..., 
         [-104.896698  , -106.50997925, -108.82247925, ...,   70.243927  ,
            81.1579895 ,   88.978302  ],
         [-105.334198  , -106.83810425, -108.98654175, ...,   64.53689575,
            71.81033325,   77.415802  ],
         [-105.771698  , -107.084198  , -108.959198  , ...,   60.790802  ,
            64.728302  ,   68.228302  ]]]], dtype=float32), 'im_info': array([[ 800.        ,  600.        ,    1.60000002]], dtype=float32)}
{'gt_boxes': array([[  65.15151215,  215.15151978,  524.24243164,  569.69696045,   26.        ],
       [ 446.96969604,  357.57574463,  757.57574463,  598.48486328,   26.        ]], dtype=float32), 'data': array([[[[ 103.01989746,   99.58989716,   90.61990356, ...,   -2.3300755 ,
             0.96994275,    2.01990008],
         [  70.67990112,   73.49250031,   70.06439972, ...,    0.33549571,
            -0.24530748,   -0.43009996],
         [  31.01990128,   39.52140045,   42.16740417, ...,    2.10487318,
            -1.52514696,   -2.68009996],
         ..., 
         [  95.76986694,   95.76986694,   96.19736481, ...,  -79.96212769,
           -84.18617249,  -85.53012085],
         [  96.54992676,   96.30001831,   96.413414  , ...,  -81.35386658,
           -85.63728333,  -87.00012207],
         [  98.01989746,   97.52989197,   97.31990051, ...,  -82.76013184,
           -86.72015381,  -87.98010254]],

        [[  88.05349731,   84.62349701,   75.65350342, ...,  -11.29647541,
            -7.9964571 ,   -6.94649982],
         [  54.73349762,   57.5460968 ,   54.11800385, ...,   -8.6309042 ,
            -9.21170807,   -9.39649963],
         [  13.90350056,   22.40500069,   25.0510025 , ...,   -7.01152658,
           -10.6415472 ,  -11.79649925],
         ..., 
         [  86.8034668 ,   86.8034668 ,   87.23097229, ...,  -76.70751953,
           -79.80954742,  -80.79650879],
         [  87.58352661,   87.33361816,   87.44702148, ...,  -77.1728363 ,
           -80.8094635 ,  -81.96652222],
         [  89.05349731,   88.56349182,   88.35350037, ...,  -77.72653198,
           -81.68655396,  -82.94650269]],

        [[  93.228302  ,   89.79829407,   80.82830048, ...,  -13.12167549,
            -9.82165718,   -8.77169991],
         [  59.41830063,   62.23090363,   58.80279922, ...,  -10.45610428,
           -11.0369072 ,  -11.22170067],
         [  17.77830124,   26.27980042,   28.92580032, ...,   -8.38672638,
           -12.01674747,  -13.17169952],
         ..., 
         [  84.97827148,   84.97827148,   85.40576935, ...,  -76.68270874,
           -79.78473663,  -80.771698  ],
         [  85.7583313 ,   85.50842285,   85.62181854, ...,  -76.99803162,
           -80.63465881,  -81.79171753],
         [  87.228302  ,   86.73830414,   86.52830505, ...,  -77.55172729,
           -81.51174927,  -82.771698  ]]]], dtype=float32), 'im_info': array([[ 600.       ,  758.       ,    1.5151515]], dtype=float32)}
{'gt_boxes': array([[   5.98591566,   87.32394409,  633.09857178,  511.97183228,   58.        ]], dtype=float32), 'data': array([[[[-13.91930008,  -8.19929981, -13.30010033, ..., -28.80412483,
          -26.07765961, -28.08595848],
         [-13.20090103,  -4.6729002 ,  -0.88409996, ..., -27.62861633,
          -27.82015991, -27.20064735],
         [ -8.71610069, -17.38809967,  -9.98010063, ..., -26.85939407,
          -24.66007996, -27.10822296],
         ..., 
         [ 24.47587967,  13.56392384,   7.57976389, ..., -69.85986328,
          -72.76400757, -58.2203331 ],
         [ 11.24070168,  12.00706863,   7.49988031, ..., -69.76400757,
          -67.20253754, -65.70031738],
         [ 12.13201427,  15.83773136,  12.22756577, ..., -64.70825958,
          -64.67768097, -67.12080383]],

        [[-11.88569927,  -7.30649996, -13.78650093, ..., -29.77052498,
          -28.66326523, -31.81235886],
         [-12.12890053, -11.34170055, -14.09049988, ..., -34.3550148 ,
          -35.39133453, -37.68704605],
         [-11.88249969, -17.98649979,  -9.54650116, ..., -30.82579803,
          -29.82648087, -32.27462769],
         ..., 
         [  8.50948143,  -0.49042082,  -3.18656278, ..., -57.58682251,
          -53.68251801, -44.98678207],
         [  1.75428128,   4.28065825,   0.77347076, ..., -63.51469421,
          -58.94649887, -58.14669418],
         [ 10.24557018,  13.47286606,  10.861166  , ..., -58.07456207,
          -63.86334229, -61.9271698 ]],

        [[-16.71089935, -14.22929955, -20.69170189, ..., -34.75572586,
          -28.46927834, -25.87755966],
         [-20.71409988, -18.74449921, -20.43569946, ..., -30.66021729,
          -28.73655319, -26.51224709],
         [-21.70769882, -28.2677002 , -19.57170105, ..., -28.65099716,
          -26.85168076, -29.29982376],
         ..., 
         [  3.68428135,  -8.37963963, -14.41178703, ..., -56.33206177,
          -46.57972717, -31.21200943],
         [ -1.31090903,  -3.5445416 , -11.0517292 , ..., -63.64382172,
          -55.13649368, -50.45187759],
         [  7.42037058,   6.6476655 ,  -0.76403391, ..., -59.8997612 ,
          -63.06925964, -61.91228485]]]], dtype=float32), 'im_info': array([[  6.00000000e+02,   8.00000000e+02,   3.52112681e-01]], dtype=float32)}
{'gt_boxes': array([[ 162.5806427 ,   15.48387051,  754.83868408,  584.51611328,  200.        ]], dtype=float32), 'data': array([[[[ 143.01989746,  143.01989746,  143.60324097, ...,  144.21572876,
           144.01989746,  144.01989746],
         [ 143.01989746,  143.01989746,  143.60324097, ...,  144.21572876,
           144.01989746,  144.01989746],
         [ 143.60324097,  143.60324097,  144.12277222, ...,  143.80375671,
           143.43656921,  143.43656921],
         ..., 
         [ 152.01989746,  152.01989746,  152.01989746, ...,  147.87406921,
           147.87406921,  147.87406921],
         [ 152.01989746,  152.01989746,  152.01989746, ...,  148.01989746,
           148.01989746,  148.01989746],
         [ 152.01989746,  152.01989746,  152.01989746, ...,  148.01989746,
           148.01989746,  148.01989746]],

        [[ 130.05349731,  130.05349731,  130.63682556, ...,  136.24932861,
           136.05349731,  136.05349731],
         [ 130.05349731,  130.05349731,  130.63682556, ...,  136.24932861,
           136.05349731,  136.05349731],
         [ 130.63682556,  130.63682556,  131.15635681, ...,  135.83735657,
           135.47016907,  135.47016907],
         ..., 
         [ 139.05349731,  139.05349731,  139.05349731, ...,  139.05349731,
           139.05349731,  139.05349731],
         [ 139.05349731,  139.05349731,  139.05349731, ...,  139.05349731,
           139.05349731,  139.05349731],
         [ 139.05349731,  139.05349731,  139.05349731, ...,  139.05349731,
           139.05349731,  139.05349731]],

        [[ 123.228302  ,  123.228302  ,  123.81163025, ...,  132.228302  ,
           132.228302  ,  132.228302  ],
         [ 123.228302  ,  123.228302  ,  123.81163025, ...,  132.228302  ,
           132.228302  ,  132.228302  ],
         [ 123.81163025,  123.81163025,  124.3311615 , ...,  131.75920105,
           131.64497375,  131.64497375],
         ..., 
         [ 132.228302  ,  132.228302  ,  132.228302  , ...,  132.228302  ,
           132.228302  ,  132.228302  ],
         [ 132.228302  ,  132.228302  ,  132.228302  , ...,  132.228302  ,
           132.228302  ,  132.228302  ],
         [ 132.228302  ,  132.228302  ,  132.228302  , ...,  132.228302  ,
           132.228302  ,  132.228302  ]]]], dtype=float32), 'im_info': array([[ 600.        ,  774.        ,    3.87096763]], dtype=float32)}
{'gt_boxes': array([[  39.62264252,   43.39622498,  903.77355957,  524.52832031,   26.        ]], dtype=float32), 'data': array([[[[-77.98010254, -80.34010315, -84.58010864, ...,  -7.45025396,
            3.42480731,  13.01989937],
         [-80.34010315, -82.2649765 , -85.72323608, ...,  -5.09025431,
            5.63878345,  15.0848999 ],
         [-84.58010864, -85.72322845, -87.77697754, ...,  -0.8502537 ,
            9.61643696,  18.79489899],
         ..., 
         [-88.13037109, -78.21076965, -60.38908768, ...,   9.65501118,
            2.1556592 ,  -6.50506353],
         [-76.47039795, -76.71351624, -77.15029907, ...,  16.62174225,
            5.38339806,  -8.09505939],
         [-69.98010254, -75.88010406, -86.48010254, ...,  20.4996357 ,
            7.18005562,  -8.98009968]],

        [[-70.94650269, -70.06150818, -68.47150421, ..., -10.59167194,
            0.4584074 ,  10.05350018],
         [-72.42150879, -71.36245728, -69.45975494, ...,  -8.23167229,
            2.67238355,  12.11849976],
         [-75.07150269, -73.69975281, -71.23525238, ...,  -3.99167204,
            6.65003538,  15.82849884],
         ..., 
         [-43.44674301, -34.95051575, -19.68610001, ...,  32.68248749,
           26.5257225 ,  19.22848701],
         [-32.84677124, -34.35691452, -37.07006073, ...,  39.63066483,
           30.802845  ,  19.75848579],
         [-26.94650078, -34.0265007 , -46.74649811, ...,  43.4982338 ,
           33.18362427,  20.05349922]],

        [[-59.771698  , -61.54170227, -64.72169495, ...,  12.40810966,
           25.61318588,  37.22829819],
         [-61.83670425, -63.25860214, -65.8132019 , ...,  14.76811028,
           27.8271637 ,  39.29330063],
         [-65.54669952, -66.34320068, -67.77420044, ...,  19.00811005,
           31.80481529,  43.00329971],
         ..., 
         [-61.62191772, -52.48406982, -36.06690979, ...,  13.50731087,
            7.09069204,  -0.47165117],
         [-52.08194351, -52.79411697, -54.07361984, ...,  19.39549255,
            9.52078056,  -2.59164619],
         [-46.77170181, -52.96670151, -64.09670258, ...,  22.67303276,
           10.8734417 ,  -3.77169991]]]], dtype=float32), 'im_info': array([[ 600.        ,  943.        ,    1.88679242]], dtype=float32)}
{'gt_boxes': array([[   0.        ,    0.        ,  100.        ,  992.30767822,  118.        ]], dtype=float32), 'data': array([[[[ 146.01989746,  146.01989746,  146.01989746, ...,  150.01989746,
           150.01989746,  150.01989746],
         [ 146.01989746,  146.01989746,  146.01989746, ...,  150.01989746,
           150.01989746,  150.01989746],
         [ 146.01989746,  146.01989746,  146.01989746, ...,  150.01989746,
           150.01989746,  150.01989746],
         ..., 
         [ 136.01989746,  136.01989746,  136.01989746, ...,  152.01989746,
           152.01989746,  152.01989746],
         [ 136.01989746,  136.01989746,  136.01989746, ...,  152.01989746,
           152.01989746,  152.01989746],
         [ 136.01989746,  136.01989746,  136.01989746, ...,  152.01989746,
           152.01989746,  152.01989746]],

        [[ 138.05349731,  138.05349731,  138.05349731, ...,  137.05349731,
           137.05349731,  137.05349731],
         [ 138.05349731,  138.05349731,  138.05349731, ...,  137.05349731,
           137.05349731,  137.05349731],
         [ 138.05349731,  138.05349731,  138.05349731, ...,  137.05349731,
           137.05349731,  137.05349731],
         ..., 
         [ 122.05349731,  122.05349731,  122.05349731, ...,  139.05349731,
           139.05349731,  139.05349731],
         [ 122.05349731,  122.053

@JohnnyY8
Copy link

@leejiajun Hi!
I noticed that I made some mistakes about the content of your blog. As your blog said, both image ration and object bounding box ratio should be within 0.117-15.500, is that right?

@leejiajun
Copy link

@JohnnyY8
They should be constrained in different ratio.

@JohnnyY8
Copy link

@leejiajun
Do you mean that the bounding box ratio should be within 0.117-15.5, but the image ratio should be other values? What is the range for image ratio ?

@leejiajun
Copy link

@JohnnyY8 I updated my blog http://blog.csdn.net/JiaJunLee/article/details/50470897

@JohnnyY8
Copy link

@leejiajun Thanks for your help!
I finished my training, the bounding box ratio in 0.117-15.500 is OK. And in pascal_voc 2007 training set, the value of minimum x, maximum x, minimum y, and maximum y are 127, 500, 96, and 500. So I remove all images without in this size.
BTW. There is a little mistakes.
image

@leejiajun
Copy link

leejiajun commented Aug 13, 2016 •
edited
Loading

@JohnnyY8
Con!
BTW. These "mistakes" are humorous words.

@JohnnyY8
Copy link

@leejiajun really thanks for your help and your time. You are so nice!

@soupault
Copy link

Removing -1 from both lib/dataset/$CUSTOM_DATASET.py:

x1 = float(get_data_from_tag(obj, 'xmin')) - 1
y1 = float(get_data_from_tag(obj, 'ymin')) - 1
x2 = float(get_data_from_tag(obj, 'xmax')) - 1
y2 = float(get_data_from_tag(obj, 'ymax')) - 1

and lib/dataset/imdb.py:

boxes[:, 0] = widths[i] - oldx2 - 1
boxes[:, 2] = widths[i] - oldx1 - 1

helped in my case.

@JohnnyY8
Copy link

@soupault Hi
It helped for me as well.

@vikiboy
Copy link

vikiboy commented Sep 26, 2016 •
edited
Loading

@Jianchao-ICT @GregorySenay how did you outline which file had empty ground truth values ?

@shubh-agrawal
Copy link

shubh-agrawal commented Sep 27, 2016 •
edited
Loading

I am trying to train py-faster-rcnn on my custom dataset. Untill now I was busy with cleaning out dataset for errors like empty gt_boxes and now am stuck with abrupt crashing of iterations at 1st one itself.
Have a look here. It crashes without any error output.
I0927 19:56:33.645869 5137 solver.cpp:245] Train net output #0: rpn_cls_loss = 0.705536 (* 1 = 0.705536 loss) I0927 19:56:33.645876 5137 solver.cpp:245] Train net output #1: rpn_loss_bbox = 0.0306727 (* 1 = 0.0306727 loss) I0927 19:56:33.645882 5137 sgd_solver.cpp:106] Iteration 0, lr = 0.001
Please give me suggestions and possible erroneous targets to resolve the issue. Thanks!

@zhyx12 zhyx12 mentioned this issue Jul 28, 2017
Open
@moyans
Copy link

moyans commented Jun 21, 2018 •
edited
Loading

@JohnnyY8 @rbgirshick @GregorySenay @sunshineatnoon @leefionglee @jojotata Where to find ZF_v2.caffemodel

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests