Getting mbox_loss as 0

[ravikantb]

Hi,
I am trying to train TextBoxes on my custom dataset. The annotations are in pascal_voc format. I am getting following log for all the iterations so far (currently it is at 830 iteration) while training TextBoxes.

Train net output #0: mbox_loss = 0 (* 1 = 0 loss)

Somehow it does not look right to me. Can someone suggest what might be wrong here. I have done a code walk through and things seem right as far as annotations are concerned. I am debugging it further but any quick help will be appreciated. Thanks.

[MhLiao]

@ravikantb Did you try the Synthetic dataset or ICDAR 2013 dataset? I suggest you try them first.

[ravikantb]

@MhLiao : Thanks for your suggestion. We (me and my friends here) tried ICDAR 2013 dataset and found what was wrong with our way of training. Apparently we were trying to train without using test_size.txt files which did not generate correct lmdb database files. Now we are able to train TextBoxes on our own dataset also :)

[ahmedmazari-dhatim]

Hi @ravikantb ,
How did you create lmdb file for your own dataset ?

[ravikantb]

Hi @ahmedmazari-dhatim ,

Following steps helped me run it on my own dataset:

1. Create list of image names using create_list.py. There are two versions of it with this code base.
   https://github.com/MhLiao/TextBoxes/blob/TextBoxes/data/coco/create_list.py
   https://github.com/MhLiao/TextBoxes/blob/TextBoxes/data/ILSVRC2016/create_list.py
   You can modify any of these for your use case.

2. Make sure to write file sizes of images in your dataset in a text file.

3. Add mapping for your custom classes in labelmap_voc.prototxt

4. Use create_data.sh file to create lmdb. Again there are three versions of it but you can modify any of them for your use case. Their core functionality is same, you just need to change some paths and configuration params. This file calls another file which eventually refers to the files created in above steps.

If you haven't already then my suggestion would be to read these create_list.py and create_data.sh (and also create_annoset.py which is called by create_data.sh) files and understand what they are doing.
If you want then I can share my files (I shall need sometime to dig them though).

Hope this helps :)

[remywilson]

@ravikantb How do I convert the dataset annotations into Pascal_VOC format (XML format) such that the codes you mentioned above can create an appropriate lmdb file, on which Textboxes could be trained successfully. Please help.

[bhargavaurala]

@MhLiao I am experiencing the same issue. I have used create_list.py and create_data.sh to create the training LMDB for ICDAR13 and my own custom dataset without any error. However both multibox loss and detection_eval is turning out to be exactly 0 for both. Could you please help figure out why this could be happening? I will share xml files of annotation if need be.

@ravikantb Could you please elaborate on step 2 in your previous comment? I am not able to see documentation to help create this text file of image sizes (is it dimensions (e.g: 1920x1080) or size on disk). I know it is more than a year ago but any hint would be helpful.

Thanks,
Bhargava

[Yinlei-lambda]

@ravikantb Hi, I have the same problem. I am trying to train SSD on COCO2017 dataset, but i generate lmdb files without train_size.txt and test_size.txt. So, is that the problem?
By the way, the value of loss starts off at 0 and stays at 0.
I am looking forward to your reply. Thank you!

[ramgo2]

I'm facing same issue.. is there someway of debbugging the issue? The logs are not useful and the error messages are not informative. In my case, increasing the size of the input images by 10% creates out-of-memory exception and without increase in size (336 x 204 is the size of the original images) I get mbox_loss = 0 (* 1 = 0 loss) messages. Don't have a clue where to look.

==================================================================
Sorry.. Again, a minor issue with annotation data completely missed by me. With the data fixed the network is training. Wish I had found this much earlier.