5cm 5 degree metric becomes zero for custom data training

[monajalal]

Could you please help how to fix this?

eta: 0:00:12  epoch: 0  step: 280  vote_loss: 0.1839  seg_loss: 0.0552  loss: 0.2391  data: 0.0541  batch: 0.3603  lr: 0.001000  max_mem: 18299
eta: 0:00:05  epoch: 0  step: 300  vote_loss: 0.1810  seg_loss: 0.0421  loss: 0.2231  data: 0.0582  batch: 0.3839  lr: 0.001000  max_mem: 18299
eta: 0:00:00  epoch: 0  step: 312  vote_loss: 0.1786  seg_loss: 0.0374  loss: 0.2160  data: 0.0533  batch: 0.3729  lr: 0.001000  max_mem: 18299
eta: 0:01:56  epoch: 1  step: 333  vote_loss: 0.1771  seg_loss: 0.0384  loss: 0.2155  data: 0.1751  batch: 0.5056  lr: 0.001000  max_mem: 18770
eta: 0:01:48  epoch: 1  step: 353  vote_loss: 0.1787  seg_loss: 0.0566  loss: 0.2353  data: 0.0585  batch: 0.3943  lr: 0.001000  max_mem: 18770
eta: 0:01:40  epoch: 1  step: 373  vote_loss: 0.1787  seg_loss: 0.0452  loss: 0.2239  data: 0.0555  batch: 0.3898  lr: 0.001000  max_mem: 18770
eta: 0:01:32  epoch: 1  step: 393  vote_loss: 0.1759  seg_loss: 0.0381  loss: 0.2140  data: 0.0583  batch: 0.3969  lr: 0.001000  max_mem: 18770
eta: 0:01:24  epoch: 1  step: 413  vote_loss: 0.1758  seg_loss: 0.0330  loss: 0.2088  data: 0.0585  batch: 0.4055  lr: 0.001000  max_mem: 18770
eta: 0:01:16  epoch: 1  step: 433  vote_loss: 0.1743  seg_loss: 0.0340  loss: 0.2083  data: 0.0559  batch: 0.3756  lr: 0.001000  max_mem: 18770
eta: 0:01:08  epoch: 1  step: 453  vote_loss: 0.1740  seg_loss: 0.0320  loss: 0.2060  data: 0.0568  batch: 0.3835  lr: 0.001000  max_mem: 18770
eta: 0:01:00  epoch: 1  step: 473  vote_loss: 0.1748  seg_loss: 0.0311  loss: 0.2059  data: 0.0562  batch: 0.3713  lr: 0.001000  max_mem: 18770
eta: 0:00:52  epoch: 1  step: 493  vote_loss: 0.1737  seg_loss: 0.0311  loss: 0.2048  data: 0.0557  batch: 0.3896  lr: 0.001000  max_mem: 18770
eta: 0:00:44  epoch: 1  step: 513  vote_loss: 0.1731  seg_loss: 0.0311  loss: 0.2041  data: 0.0548  batch: 0.3678  lr: 0.001000  max_mem: 18770
eta: 0:00:36  epoch: 1  step: 533  vote_loss: 0.1743  seg_loss: 0.0318  loss: 0.2062  data: 0.0517  batch: 0.3607  lr: 0.001000  max_mem: 18770
eta: 0:00:28  epoch: 1  step: 553  vote_loss: 0.1732  seg_loss: 0.0275  loss: 0.2007  data: 0.0584  batch: 0.3858  lr: 0.001000  max_mem: 18770
eta: 0:00:20  epoch: 1  step: 573  vote_loss: 0.1728  seg_loss: 0.0300  loss: 0.2029  data: 0.0520  batch: 0.3440  lr: 0.001000  max_mem: 18770
eta: 0:00:12  epoch: 1  step: 593  vote_loss: 0.1740  seg_loss: 0.0276  loss: 0.2016  data: 0.0583  batch: 0.3975  lr: 0.001000  max_mem: 18770
eta: 0:00:05  epoch: 1  step: 613  vote_loss: 0.1730  seg_loss: 0.0288  loss: 0.2018  data: 0.0547  batch: 0.3729  lr: 0.001000  max_mem: 18770
eta: 0:00:00  epoch: 1  step: 625  vote_loss: 0.1729  seg_loss: 0.0309  loss: 0.2038  data: 0.0498  batch: 0.3677  lr: 0.001000  max_mem: 18770
eta: 0:01:54  epoch: 2  step: 646  vote_loss: 0.1732  seg_loss: 0.0306  loss: 0.2038  data: 0.1416  batch: 0.4815  lr: 0.001000  max_mem: 18770
eta: 0:01:46  epoch: 2  step: 666  vote_loss: 0.1735  seg_loss: 0.0285  loss: 0.2020  data: 0.0548  batch: 0.3683  lr: 0.001000  max_mem: 18770
eta: 0:01:39  epoch: 2  step: 686  vote_loss: 0.1705  seg_loss: 0.0246  loss: 0.1952  data: 0.0604  batch: 0.4121  lr: 0.001000  max_mem: 18770
eta: 0:01:30  epoch: 2  step: 706  vote_loss: 0.1744  seg_loss: 0.0279  loss: 0.2022  data: 0.0534  batch: 0.3564  lr: 0.001000  max_mem: 18770
eta: 0:01:23  epoch: 2  step: 726  vote_loss: 0.1729  seg_loss: 0.0250  loss: 0.1979  data: 0.0551  batch: 0.3764  lr: 0.001000  max_mem: 18770
eta: 0:01:15  epoch: 2  step: 746  vote_loss: 0.1724  seg_loss: 0.0256  loss: 0.1980  data: 0.0586  batch: 0.3987  lr: 0.001000  max_mem: 18770
eta: 0:01:07  epoch: 2  step: 766  vote_loss: 0.1716  seg_loss: 0.0266  loss: 0.1982  data: 0.0513  batch: 0.3595  lr: 0.001000  max_mem: 18770
eta: 0:00:59  epoch: 2  step: 786  vote_loss: 0.1731  seg_loss: 0.0264  loss: 0.1994  data: 0.0592  batch: 0.4048  lr: 0.001000  max_mem: 18881
eta: 0:00:51  epoch: 2  step: 806  vote_loss: 0.1713  seg_loss: 0.0238  loss: 0.1950  data: 0.0580  batch: 0.4127  lr: 0.001000  max_mem: 18881
eta: 0:00:44  epoch: 2  step: 826  vote_loss: 0.1705  seg_loss: 0.0226  loss: 0.1931  data: 0.0621  batch: 0.4275  lr: 0.001000  max_mem: 18881
eta: 0:00:36  epoch: 2  step: 846  vote_loss: 0.1713  seg_loss: 0.0241  loss: 0.1954  data: 0.0578  batch: 0.4055  lr: 0.001000  max_mem: 18881
eta: 0:00:28  epoch: 2  step: 866  vote_loss: 0.1721  seg_loss: 0.0268  loss: 0.1990  data: 0.0490  batch: 0.3489  lr: 0.001000  max_mem: 18881
eta: 0:00:20  epoch: 2  step: 886  vote_loss: 0.1712  seg_loss: 0.0278  loss: 0.1991  data: 0.0504  batch: 0.3630  lr: 0.001000  max_mem: 18881
eta: 0:00:12  epoch: 2  step: 906  vote_loss: 0.1699  seg_loss: 0.0250  loss: 0.1949  data: 0.0543  batch: 0.3726  lr: 0.001000  max_mem: 18881
eta: 0:00:05  epoch: 2  step: 926  vote_loss: 0.1710  seg_loss: 0.0244  loss: 0.1954  data: 0.0573  batch: 0.4046  lr: 0.001000  max_mem: 18881
eta: 0:00:00  epoch: 2  step: 938  vote_loss: 0.1696  seg_loss: 0.0251  loss: 0.1947  data: 0.0533  batch: 0.4125  lr: 0.001000  max_mem: 19135
eta: 0:01:55  epoch: 3  step: 959  vote_loss: 0.1706  seg_loss: 0.0236  loss: 0.1941  data: 0.1770  batch: 0.5701  lr: 0.001000  max_mem: 19438
eta: 0:01:47  epoch: 3  step: 979  vote_loss: 0.1717  seg_loss: 0.0269  loss: 0.1986  data: 0.0523  batch: 0.3738  lr: 0.001000  max_mem: 19438
eta: 0:01:39  epoch: 3  step: 999  vote_loss: 0.1667  seg_loss: 0.0243  loss: 0.1910  data: 0.0596  batch: 0.4081  lr: 0.001000  max_mem: 19438
eta: 0:01:32  epoch: 3  step: 1019  vote_loss: 0.1684  seg_loss: 0.0294  loss: 0.1978  data: 0.0656  batch: 0.4562  lr: 0.001000  max_mem: 19438
eta: 0:01:24  epoch: 3  step: 1039  vote_loss: 0.1748  seg_loss: 0.0481  loss: 0.2229  data: 0.0589  batch: 0.4043  lr: 0.001000  max_mem: 19438
eta: 0:01:16  epoch: 3  step: 1059  vote_loss: 0.1755  seg_loss: 0.0461  loss: 0.2216  data: 0.0613  batch: 0.4024  lr: 0.001000  max_mem: 19438
eta: 0:01:08  epoch: 3  step: 1079  vote_loss: 0.1726  seg_loss: 0.0393  loss: 0.2119  data: 0.0623  batch: 0.4182  lr: 0.001000  max_mem: 19438
eta: 0:01:00  epoch: 3  step: 1099  vote_loss: 0.1627  seg_loss: 0.0319  loss: 0.1946  data: 0.0625  batch: 0.4246  lr: 0.001000  max_mem: 19438
eta: 0:00:52  epoch: 3  step: 1119  vote_loss: 0.1638  seg_loss: 0.0330  loss: 0.1968  data: 0.0527  batch: 0.3623  lr: 0.001000  max_mem: 19438
eta: 0:00:44  epoch: 3  step: 1139  vote_loss: 0.1666  seg_loss: 0.0303  loss: 0.1969  data: 0.0570  batch: 0.3689  lr: 0.001000  max_mem: 19438
eta: 0:00:36  epoch: 3  step: 1159  vote_loss: 0.1621  seg_loss: 0.0372  loss: 0.1993  data: 0.0598  batch: 0.4063  lr: 0.001000  max_mem: 19438
eta: 0:00:28  epoch: 3  step: 1179  vote_loss: 0.1615  seg_loss: 0.0397  loss: 0.2012  data: 0.0479  batch: 0.3461  lr: 0.001000  max_mem: 19438
eta: 0:00:20  epoch: 3  step: 1199  vote_loss: 0.1490  seg_loss: 0.0318  loss: 0.1808  data: 0.0574  batch: 0.3985  lr: 0.001000  max_mem: 19438
eta: 0:00:13  epoch: 3  step: 1219  vote_loss: 0.1434  seg_loss: 0.0296  loss: 0.1730  data: 0.0556  batch: 0.3899  lr: 0.001000  max_mem: 19438
eta: 0:00:05  epoch: 3  step: 1239  vote_loss: 0.1393  seg_loss: 0.0299  loss: 0.1692  data: 0.0593  batch: 0.4036  lr: 0.001000  max_mem: 19438
eta: 0:00:00  epoch: 3  step: 1251  vote_loss: 0.1304  seg_loss: 0.0321  loss: 0.1625  data: 0.0540  batch: 0.3822  lr: 0.001000  max_mem: 19438
eta: 0:01:55  epoch: 4  step: 1272  vote_loss: 0.1304  seg_loss: 0.0336  loss: 0.1640  data: 0.1529  batch: 0.4858  lr: 0.001000  max_mem: 19438
eta: 0:01:48  epoch: 4  step: 1292  vote_loss: 0.1141  seg_loss: 0.0336  loss: 0.1477  data: 0.0600  batch: 0.4106  lr: 0.001000  max_mem: 19438
eta: 0:01:40  epoch: 4  step: 1312  vote_loss: 0.1229  seg_loss: 0.0354  loss: 0.1583  data: 0.0564  batch: 0.3718  lr: 0.001000  max_mem: 19438
eta: 0:01:32  epoch: 4  step: 1332  vote_loss: 0.1351  seg_loss: 0.0360  loss: 0.1711  data: 0.0546  batch: 0.3740  lr: 0.001000  max_mem: 19438
eta: 0:01:24  epoch: 4  step: 1352  vote_loss: 0.1108  seg_loss: 0.0374  loss: 0.1482  data: 0.0580  batch: 0.3982  lr: 0.001000  max_mem: 19438
eta: 0:01:16  epoch: 4  step: 1372  vote_loss: 0.0983  seg_loss: 0.0439  loss: 0.1423  data: 0.0540  batch: 0.3656  lr: 0.001000  max_mem: 19438
eta: 0:01:08  epoch: 4  step: 1392  vote_loss: 0.1073  seg_loss: 0.0370  loss: 0.1443  data: 0.0556  batch: 0.3838  lr: 0.001000  max_mem: 19438
eta: 0:01:00  epoch: 4  step: 1412  vote_loss: 0.0987  seg_loss: 0.0323  loss: 0.1310  data: 0.0591  batch: 0.4103  lr: 0.001000  max_mem: 19438
eta: 0:00:52  epoch: 4  step: 1432  vote_loss: 0.0870  seg_loss: 0.0320  loss: 0.1190  data: 0.0567  batch: 0.3936  lr: 0.001000  max_mem: 19438
eta: 0:00:44  epoch: 4  step: 1452  vote_loss: 0.0958  seg_loss: 0.0334  loss: 0.1292  data: 0.0565  batch: 0.3714  lr: 0.001000  max_mem: 19438
eta: 0:00:36  epoch: 4  step: 1472  vote_loss: 0.0921  seg_loss: 0.0313  loss: 0.1234  data: 0.0601  batch: 0.4310  lr: 0.001000  max_mem: 19438
eta: 0:00:28  epoch: 4  step: 1492  vote_loss: 0.0848  seg_loss: 0.0362  loss: 0.1210  data: 0.0530  batch: 0.3556  lr: 0.001000  max_mem: 19438
eta: 0:00:20  epoch: 4  step: 1512  vote_loss: 0.0719  seg_loss: 0.0321  loss: 0.1040  data: 0.0553  batch: 0.3820  lr: 0.001000  max_mem: 19438
eta: 0:00:13  epoch: 4  step: 1532  vote_loss: 0.0673  seg_loss: 0.0302  loss: 0.0975  data: 0.0543  batch: 0.3734  lr: 0.001000  max_mem: 19438
eta: 0:00:05  epoch: 4  step: 1552  vote_loss: 0.0760  seg_loss: 0.0309  loss: 0.1069  data: 0.0577  batch: 0.3989  lr: 0.001000  max_mem: 19438
eta: 0:00:00  epoch: 4  step: 1564  vote_loss: 0.0724  seg_loss: 0.0324  loss: 0.1048  data: 0.0516  batch: 0.3723  lr: 0.001000  max_mem: 19438
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9999/9999 [04:48<00:00, 34.63it/s]
['vote_loss: 0.0635', 'seg_loss: 0.0175', 'loss: 0.0810']
2d projections metric: 0.0009000900090009
ADD metric: 0.0333033303330333
5 cm 5 degree metric: 0.0
mask ap70: 0.9965996599659966

[utsavrai]

Hi @monajalal, did you find any resolution to your problem, I am facing the same issue as I train on my custom data the 5 cm 5 degree metric remains zero

[Mechazo11]

Hi @monajalal I saw another issue you opened earlier about training a custom dataset. Can you share the steps on how you created a custom dataset?