Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cann't overfitt SparseInst #34

Open
sdimantsd opened this issue May 11, 2022 · 1 comment
Open

Cann't overfitt SparseInst #34

sdimantsd opened this issue May 11, 2022 · 1 comment

Comments

@sdimantsd
Copy link

sdimantsd commented May 11, 2022 •
edited
Loading

Hi,
I am try to overfit one image using sparse_inst_r50_giam config.
I changed the dataset to a custom dataset and only 3 labels (car, bus, truck).
Those are the lines in the log (after one day of training):

[05/11 12:29:09 d2.utils.events]:  eta: 9 days, 2:03:30  iter: 119739  total_loss: 0.873  loss_box: 0.4872  loss_obj_pos: 0.0006603  loss_obj_neg: 0.003101  loss_cls: 0.04907  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.128e-07  loss_wh: 0.02689  time: 0.5578  data_time: 0.0603  lr: 0.00011079  max_mem: 4517M
[05/11 12:29:20 d2.utils.events]:  eta: 9 days, 2:06:10  iter: 119759  total_loss: 0.8729  loss_box: 0.4872  loss_obj_pos: 0.0006603  loss_obj_neg: 0.0031  loss_cls: 0.04907  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.165e-07  loss_wh: 0.02699  time: 0.5578  data_time: 0.0607  lr: 0.00011079  max_mem: 4517M
[05/11 12:29:31 d2.utils.events]:  eta: 9 days, 2:03:08  iter: 119779  total_loss: 0.8728  loss_box: 0.4871  loss_obj_pos: 0.0006602  loss_obj_neg: 0.003099  loss_cls: 0.04906  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.095e-07  loss_wh: 0.02677  time: 0.5578  data_time: 0.0594  lr: 0.00011079  max_mem: 4517M
[05/11 12:29:42 d2.utils.events]:  eta: 9 days, 2:05:47  iter: 119799  total_loss: 0.8728  loss_box: 0.4871  loss_obj_pos: 0.0006602  loss_obj_neg: 0.003099  loss_cls: 0.04906  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.129e-07  loss_wh: 0.02687  time: 0.5578  data_time: 0.0605  lr: 0.00011078  max_mem: 4517M
[05/11 12:29:54 d2.utils.events]:  eta: 9 days, 2:08:15  iter: 119819  total_loss: 0.8727  loss_box: 0.487  loss_obj_pos: 0.0006601  loss_obj_neg: 0.003098  loss_cls: 0.04905  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.164e-07  loss_wh: 0.02697  time: 0.5578  data_time: 0.0610  lr: 0.00011078  max_mem: 4517M
[05/11 12:30:05 d2.utils.events]:  eta: 9 days, 2:02:34  iter: 119839  total_loss: 0.8727  loss_box: 0.487  loss_obj_pos: 0.00066  loss_obj_neg: 0.003097  loss_cls: 0.04905  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.093e-07  loss_wh: 0.02675  time: 0.5578  data_time: 0.0596  lr: 0.00011078  max_mem: 4517M
[05/11 12:30:16 d2.utils.events]:  eta: 9 days, 1:58:52  iter: 119859  total_loss: 0.8726  loss_box: 0.4869  loss_obj_pos: 0.00066  loss_obj_neg: 0.003097  loss_cls: 0.04904  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.128e-07  loss_wh: 0.02685  time: 0.5578  data_time: 0.0615  lr: 0.00011078  max_mem: 4517M
[05/11 12:30:27 d2.utils.events]:  eta: 9 days, 1:56:41  iter: 119879  total_loss: 0.8726  loss_box: 0.4869  loss_obj_pos: 0.0006598  loss_obj_neg: 0.003096  loss_cls: 0.04904  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.166e-07  loss_wh: 0.02696  time: 0.5578  data_time: 0.0604  lr: 0.00011078  max_mem: 4517M
[05/11 12:30:38 d2.utils.events]:  eta: 9 days, 1:52:10  iter: 119899  total_loss: 0.8725  loss_box: 0.4869  loss_obj_pos: 0.0006598  loss_obj_neg: 0.003096  loss_cls: 0.04903  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.093e-07  loss_wh: 0.02674  time: 0.5578  data_time: 0.0613  lr: 0.00011078  max_mem: 4517M
[05/11 12:30:49 d2.utils.events]:  eta: 9 days, 1:50:12  iter: 119919  total_loss: 0.8725  loss_box: 0.4868  loss_obj_pos: 0.0006597  loss_obj_neg: 0.003095  loss_cls: 0.04902  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.127e-07  loss_wh: 0.02684  time: 0.5578  data_time: 0.0609  lr: 0.00011078  max_mem: 4517M
[05/11 12:31:00 d2.utils.events]:  eta: 9 days, 1:43:43  iter: 119939  total_loss: 0.8724  loss_box: 0.4868  loss_obj_pos: 0.0006596  loss_obj_neg: 0.003094  loss_cls: 0.04902  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.166e-07  loss_wh: 0.02694  time: 0.5578  data_time: 0.0605  lr: 0.00011078  max_mem: 4517M
[05/11 12:31:12 d2.utils.events]:  eta: 9 days, 1:42:02  iter: 119959  total_loss: 0.8724  loss_box: 0.4867  loss_obj_pos: 0.0006595  loss_obj_neg: 0.003094  loss_cls: 0.04901  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.092e-07  loss_wh: 0.02672  time: 0.5578  data_time: 0.0610  lr: 0.00011078  max_mem: 4517M
[05/11 12:31:23 d2.utils.events]:  eta: 9 days, 1:40:19  iter: 119979  total_loss: 0.8723  loss_box: 0.4867  loss_obj_pos: 0.0006595  loss_obj_neg: 0.003093  loss_cls: 0.04901  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.128e-07  loss_wh: 0.02682  time: 0.5578  data_time: 0.0623  lr: 0.00011078  max_mem: 4517M
[05/11 12:31:34 fvcore.common.checkpoint]: Saving checkpoint to output/coco_yolomask/model_0119999.pth
[05/11 12:31:34 d2.utils.events]:  eta: 9 days, 1:27:01  iter: 119999  total_loss: 0.8723  loss_box: 0.4866  loss_obj_pos: 0.0006594  loss_obj_neg: 0.003092  loss_cls: 0.049  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.162e-07  loss_wh: 0.02693  time: 0.5578  data_time: 0.0613  lr: 0.00011078  max_mem: 4517M
[05/11 12:31:45 d2.utils.events]:  eta: 9 days, 1:24:30  iter: 120019  total_loss: 0.8722  loss_box: 0.4866  loss_obj_pos: 0.0006593  loss_obj_neg: 0.003091  loss_cls: 0.049  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.089e-07  loss_wh: 0.02671  time: 0.5578  data_time: 0.0606  lr: 0.00011078  max_mem: 4517M
[05/11 12:31:57 d2.utils.events]:  eta: 9 days, 1:18:55  iter: 120039  total_loss: 0.8721  loss_box: 0.4866  loss_obj_pos: 0.0006592  loss_obj_neg: 0.00309  loss_cls: 0.04899  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.122e-07  loss_wh: 0.02681  time: 0.5578  data_time: 0.0609  lr: 0.00011078  max_mem: 4517M
[05/11 12:32:08 d2.utils.events]:  eta: 9 days, 1:22:10  iter: 120059  total_loss: 0.8721  loss_box: 0.4865  loss_obj_pos: 0.0006591  loss_obj_neg: 0.00309  loss_cls: 0.04899  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.161e-07  loss_wh: 0.02691  time: 0.5578  data_time: 0.0604  lr: 0.00011078  max_mem: 4517M
[05/11 12:32:19 d2.utils.events]:  eta: 9 days, 1:21:07  iter: 120079  total_loss: 0.872  loss_box: 0.4865  loss_obj_pos: 0.0006591  loss_obj_neg: 0.003089  loss_cls: 0.04898  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.092e-07  loss_wh: 0.02669  time: 0.5578  data_time: 0.0622  lr: 0.00011078  max_mem: 4517M
[05/11 12:32:30 d2.utils.events]:  eta: 9 days, 1:23:45  iter: 120099  total_loss: 0.872  loss_box: 0.4864  loss_obj_pos: 0.0006591  loss_obj_neg: 0.003089  loss_cls: 0.04898  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.121e-07  loss_wh: 0.02679  time: 0.5578  data_time: 0.0615  lr: 0.00011078  max_mem: 4517M
[05/11 12:32:41 d2.utils.events]:  eta: 9 days, 1:20:48  iter: 120119  total_loss: 0.8719  loss_box: 0.4864  loss_obj_pos: 0.000659  loss_obj_neg: 0.003088  loss_cls: 0.04897  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.158e-07  loss_wh: 0.02689  time: 0.5578  data_time: 0.0607  lr: 0.00011078  max_mem: 4517M
[05/11 12:32:53 d2.utils.events]:  eta: 9 days, 1:13:26  iter: 120139  total_loss: 0.8719  loss_box: 0.4863  loss_obj_pos: 0.000659  loss_obj_neg: 0.003088  loss_cls: 0.04896  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.09e-07  loss_wh: 0.02668  time: 0.5578  data_time: 0.0608  lr: 0.00011077  max_mem: 4517M
[05/11 12:33:04 d2.utils.events]:  eta: 9 days, 1:08:34  iter: 120159  total_loss: 0.8718  loss_box: 0.4863  loss_obj_pos: 0.0006589  loss_obj_neg: 0.003087  loss_cls: 0.04896  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.123e-07  loss_wh: 0.02678  time: 0.5578  data_time: 0.0610  lr: 0.00011077  max_mem: 4517M
[05/11 12:33:15 d2.utils.events]:  eta: 9 days, 1:10:41  iter: 120179  total_loss: 0.8718  loss_box: 0.4863  loss_obj_pos: 0.0006589  loss_obj_neg: 0.003087  loss_cls: 0.04896  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.162e-07  loss_wh: 0.02688  time: 0.5578  data_time: 0.0616  lr: 0.00011077  max_mem: 4517M
[05/11 12:33:26 d2.utils.events]:  eta: 9 days, 1:20:52  iter: 120199  total_loss: 0.8717  loss_box: 0.4862  loss_obj_pos: 0.0006589  loss_obj_neg: 0.003086  loss_cls: 0.04895  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.086e-07  loss_wh: 0.02666  time: 0.5578  data_time: 0.0604  lr: 0.00011077  max_mem: 4517M
[05/11 12:33:37 d2.utils.events]:  eta: 9 days, 1:17:36  iter: 120219  total_loss: 0.8716  loss_box: 0.4862  loss_obj_pos: 0.0006589  loss_obj_neg: 0.003085  loss_cls: 0.04894  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.121e-07  loss_wh: 0.02676  time: 0.5578  data_time: 0.0617  lr: 0.00011077  max_mem: 4517M
[05/11 12:33:48 d2.utils.events]:  eta: 9 days, 1:07:31  iter: 120239  total_loss: 0.8716  loss_box: 0.4861  loss_obj_pos: 0.0006588  loss_obj_neg: 0.003085  loss_cls: 0.04894  loss_orien_pos: 0.1568  loss_orien_neg: 0.1485  loss_xy: 1.154e-07  loss_wh: 0.02686  time: 0.5578  data_time: 0.0609  lr: 0.00011077  max_mem: 4517M
[05/11 12:33:59 d2.utils.events]:  eta: 9 days, 1:11:03  iter: 120259  total_loss: 0.8715  loss_box: 0.4861  loss_obj_pos: 0.0006588  loss_obj_neg: 0.003084  loss_cls: 0.04893  loss_orien_pos: 0.1569  loss_orien_neg: 0.1485  loss_xy: 1.086e-07  loss_wh: 0.02665  time: 0.5578  data_time: 0.0598  lr: 0.00011077  max_mem: 4517M

This is the image (after one day of trainin
0_2
g):

Why the overfit did not works?
(When I used Yolact the overfit works after a 4-5 hours)

Thanks
@lucasjinreal
Copy link
Owner

I don't think this is not fit. It actually learned something. You need carefully check your lr, gama, steps, and even change the optimizer suite for your tiny dataset.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lucasjinreal @sdimantsd