Problem during training

[peternara]

Hi, @argman
I get the following error during training:

,,,
Step 000830, model loss 0.0111, total loss 0.0264, 71.25 seconds/step, 0.39 examples/second
Step 000840, model loss 0.0121, total loss 0.0272, 71.00 seconds/step, 0.39 examples/second
Step 000850, model loss 0.0124, total loss 0.0274, 71.36 seconds/step, 0.39 examples/second
Step 000860, model loss 0.0130, total loss 0.0279, 71.22 seconds/step, 0.39 examples/second
Step 000870, model loss 0.0107, total loss 0.0255, 71.07 seconds/step, 0.39 examples/second
Step 000880, model loss 0.0109, total loss 0.0256, 70.99 seconds/step, 0.39 examples/second
StepTraceback (most recent call last):
  File ".../EAST/icdar.py", line 657, in generator
    score_map, geo_map, training_mask = generate_rbox((new_h, new_w), text_polys, text_tags)
  File ".../EAST/icdar.py", line 520, in generate_rbox
    if point_dist_to_line(p1, new_p2, p0) > point_dist_to_line(p1, new_p2, p3):
  File ".../EAST/icdar.py", line 248, in point_dist_to_line
    return np.linalg.norm(np.cross(p2 - p1, p1 - p3)) / np.linalg.norm(p2 - p1)
TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'

but, training proceeds without stopping.
Do you know anything about this problem? And is it a serious problem in model creation?

[argman]

I think its caused by some un-regular polygons(mayebe unconvex or too small), you can print out the polygon and check it. And i dont find it serious.

[peternara]

@argman Thank you for your comment.
Despite the existence of the above error, the model was created.
The simple test results using it are not good. In other words, it was not better than your model.

I just used only ICDAR 2015 dataset.
Have you used ICDAR 2015/2013 dataset at the same time (for training)?

[argman]

Yes, i think 2013 will be about 2~3 points better. and I train the model with 4 gpus.

[ghost]

@peternara
Hi, I also met this problem.

It always appears during the dataset generator. I use ICDAR2013. How do you solve this problem? Could you provide some info?

Thanks a lot.

[peternara]

@argman thanks for your comment.
currently, I can not get downloads for 2013 dataset. :( Can you share the data?
Additionally, I am looking for similar data (with rotation information).

Anyway, thank you for providing the source.
I will continue testing. If you have any questions, you will be asked again.:)