Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

the error on train.py when the iteration=120170 #1

Closed
xlm998 opened this issue May 7, 2019 · 4 comments
Closed

the error on train.py when the iteration=120170 #1

xlm998 opened this issue May 7, 2019 · 4 comments

Comments

@xlm998
Copy link

xlm998 commented May 7, 2019

when the iteration=120170,this error was occured and the loss is not too small .
as follows:
epoch : 231 iter : 119990 step : 332 loss : 2.7521151838631464
epoch : 231 iter : 120000 step : 342 loss : 2.3202000970709813
epoch : 231 iter : 120010 step : 352 loss : 2.2625636844844608
epoch : 231 iter : 120020 step : 362 loss : 2.1100341796875
epoch : 231 iter : 120030 step : 372 loss : 2.1073843819754465
epoch : 231 iter : 120040 step : 382 loss : 2.1795616149902344
epoch : 231 iter : 120050 step : 392 loss : 2.4456642951754577
epoch : 231 iter : 120060 step : 402 loss : 2.0380105777662627
epoch : 231 iter : 120070 step : 412 loss : 2.336949090537785
epoch : 231 iter : 120080 step : 422 loss : 2.598242623465402
epoch : 231 iter : 120090 step : 432 loss : 2.424121765863328
epoch : 231 iter : 120100 step : 442 loss : 2.4087103054608185
epoch : 231 iter : 120110 step : 452 loss : 2.2298266662293997
epoch : 231 iter : 120120 step : 462 loss : 1.706140349893009
epoch : 231 iter : 120130 step : 472 loss : 2.384180556248574
epoch : 231 iter : 120140 step : 482 loss : 2.154784760944584
epoch : 231 iter : 120150 step : 492 loss : 2.1820303144909086
epoch : 231 iter : 120160 step : 502 loss : 2.0698028564453126
epoch : 231 iter : 120170 step : 512 loss : 2.167848598457382
(M2Det) xlm@xlm-OptiPlex-9010:/ssd_pytorch-master$ 000.
000.: command not found
(M2Det) xlm@xlm-OptiPlex-9010:
/ssd_pytorch-master$
(M2Det) xlm@xlm-OptiPlex-9010:/ssd_pytorch-master$
(M2Det) xlm@xlm-OptiPlex-9010:
/ssd_pytorch-master$

@acm5656
Copy link
Owner

acm5656 commented May 7, 2019

I think you can test whether the object can be detected by using the weight of 120000 rounds of training. The loss in my training is also around 2, and if you want to obtain lower loss, you can consider reducing the learning rate.

@xlm998
Copy link
Author

xlm998 commented May 7, 2019

how to test not a single image ,and how to eval this train and test result,such as AP and total recall?can you help me?

@acm5656
Copy link
Owner

acm5656 commented May 7, 2019

I'm sorry that this version doesn't provide the method to eval train and test result. I will complete this function in my spare time.

@xlm998
Copy link
Author

xlm998 commented May 7, 2019

thank you for you answer,when use the ssd-pytorch-master on https://github.com/amdegroot/ssd.pytorch, there is a error in training

iter 900 || Loss: 6.6272 || timer: 0.1010 sec.
iter 910 || Loss: 7.0335 || timer: 0.1023 sec.
iter 920 || Loss: 6.6000 || timer: 0.1001 sec.
iter 930 || Loss: 6.7137 || timer: 0.1013 sec.
iter 940 || Loss: 6.9450 || timer: 0.1027 sec.
iter 950 || Loss: 6.5815 || timer: 0.1038 sec.
iter 960 || Loss: 6.8804 || timer: 0.1021 sec.
iter 970 || Loss: 6.6749 || timer: 0.1279 sec.
iter 980 || Loss: 6.4802 || timer: 0.1018 sec.
iter 990 || Loss: 6.1978 || timer: 0.1184 sec.
iter 1000 || Loss: 6.7934 || timer: 0.1019 sec.
iter 1010 || Loss: 6.5664 || timer: 0.1028 sec.
iter 1020 || Loss: 6.6167 || timer: 0.0977 sec.
iter 1030 || Loss: 6.3809 || Traceback (most recent call last):
File "train.py", line 261, in
train()
File "train.py", line 166, in train
images, targets = next(batch_iterator)
File "/home/xlm/anaconda3/envs/M2Det/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 627, in next
raise StopIteration
StopIteration

when i follow the issues in https://github.com/amdegroot/ssd.pytorch

change
images, targets = next(batch_iterator)
with
try:
images,targets = next(batch_iterator)
except StopIteration:
bath_interator = iter(data_loader)
images,targets = next(batch_iterator)
but this error was still occured.

can you help me to address this problem

@xlm998 xlm998 closed this as completed May 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants