validation error

[sawwhite]

I used the command "python validate.py /imagenet/validation/ --model seresnext26_32x4d --pretrained"

however I get results as follows:
Model seresnext26_32x4d created, param count: 16790280
Data processing configuration for current model + dataset:
input_size: (3, 224, 224)
interpolation: bicubic
mean: (0.485, 0.456, 0.406)
std: (0.229, 0.224, 0.225)
crop_pct: 0.875
Test: [ 0/196] Time: 6.305s (6.305s, 40.61/s) Loss: 9.8692 (9.8692) Acc@1: 0.000 ( 0.000) Acc@5: 0.000 ( 0.000)
Test: [ 10/196] Time: 0.460s (0.993s, 257.93/s) Loss: 9.9665 (9.8110) Acc@1: 0.000 ( 0.036) Acc@5: 0.000 ( 0.178)
Test: [ 20/196] Time: 0.460s (0.738s, 346.65/s) Loss: 9.9139 (9.8199) Acc@1: 0.000 ( 0.037) Acc@5: 0.000 ( 0.279)
Test: [ 30/196] Time: 0.658s (0.690s, 371.21/s) Loss: 9.9194 (9.8095) Acc@1: 0.000 ( 0.050) Acc@5: 0.000 ( 0.265)
Test: [ 40/196] Time: 0.456s (0.676s, 378.46/s) Loss: 9.7059 (9.8126) Acc@1: 0.000 ( 0.086) Acc@5: 0.000 ( 0.286)
Test: [ 50/196] Time: 0.598s (0.659s, 388.64/s) Loss: 9.9907 (9.8099) Acc@1: 0.000 ( 0.092) Acc@5: 0.391 ( 0.306)
Test: [ 60/196] Time: 0.456s (0.658s, 388.94/s) Loss: 9.8720 (9.8085) Acc@1: 0.000 ( 0.083) Acc@5: 0.391 ( 0.327)
Test: [ 70/196] Time: 0.458s (0.647s, 395.40/s) Loss: 9.8078 (9.8059) Acc@1: 0.391 ( 0.099) Acc@5: 0.391 ( 0.330)
Test: [ 80/196] Time: 0.457s (0.650s, 394.14/s) Loss: 9.7700 (9.8087) Acc@1: 0.000 ( 0.092) Acc@5: 0.391 ( 0.333)
Test: [ 90/196] Time: 0.462s (0.641s, 399.14/s) Loss: 9.7730 (9.8074) Acc@1: 0.000 ( 0.086) Acc@5: 0.391 ( 0.331)
Test: [ 100/196] Time: 0.455s (0.636s, 402.24/s) Loss: 9.7435 (9.8020) Acc@1: 0.000 ( 0.097) Acc@5: 1.172 ( 0.340)
Test: [ 110/196] Time: 1.086s (0.636s, 402.57/s) Loss: 9.8433 (9.8015) Acc@1: 0.000 ( 0.099) Acc@5: 0.391 ( 0.338)
Test: [ 120/196] Time: 0.456s (0.634s, 403.84/s) Loss: 9.7374 (9.8034) Acc@1: 0.391 ( 0.097) Acc@5: 0.391 ( 0.339)
Test: [ 130/196] Time: 0.756s (0.633s, 404.63/s) Loss: 9.8264 (9.8040) Acc@1: 0.000 ( 0.095) Acc@5: 0.000 ( 0.337)
Test: [ 140/196] Time: 0.456s (0.631s, 405.95/s) Loss: 9.9303 (9.8048) Acc@1: 0.000 ( 0.091) Acc@5: 1.172 ( 0.346)
Test: [ 150/196] Time: 1.038s (0.631s, 405.56/s) Loss: 9.7774 (9.8034) Acc@1: 0.000 ( 0.096) Acc@5: 1.172 ( 0.360)
Test: [ 160/196] Time: 0.456s (0.630s, 406.41/s) Loss: 9.7983 (9.8024) Acc@1: 0.000 ( 0.097) Acc@5: 0.000 ( 0.352)
Test: [ 170/196] Time: 0.948s (0.628s, 407.72/s) Loss: 9.8538 (9.8048) Acc@1: 0.000 ( 0.094) Acc@5: 0.391 ( 0.345)
Test: [ 180/196] Time: 0.456s (0.625s, 409.81/s) Loss: 9.6598 (9.8046) Acc@1: 0.391 ( 0.095) Acc@5: 1.172 ( 0.354)
Test: [ 190/196] Time: 1.276s (0.630s, 406.42/s) Loss: 9.7359 (9.8046) Acc@1: 0.000 ( 0.094) Acc@5: 0.781 ( 0.360)

• Acc@1 0.094 (99.906) Acc@5 0.360 (99.640)

[AlexeyAB]

@rwightman @sawwhite Hi,

I have the same issue.

How to validate default pre-trained model resnet50?

I get low accuracy < 1% for resnet50 pre-trained model.

• I use master branch commit Commits on Nov 13, 2020 80cd31f

• I have ILSVRC2012_val dataset with 50 000 images: https://drive.google.com/file/d/117fMU8dxSIvpOzzgHhb6aSYlfM7fdHTT/view?usp=sharing

• And I did validation on both Windows7 x64 (Pytorch 1.7.0+cu110) and Linux 18.04 (Pytorch 1.6.0+cu101) as it is described there: https://rwightman.github.io/pytorch-image-models/scripts/

On Windows Pytorch is installed by using:
pip install torch===1.7.0+cu110 torchvision===0.8.1+cu110 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

---

1. On Linux (Pytorch 1.6.0):

python validate.py /export/work/alexeyab84/imagenet/validation/ --model resnet50 --pretrained

Result:

Model resnet50 created, param count: 25557032
Data processing configuration for current model + dataset:
	input_size: (3, 224, 224)
	interpolation: bicubic
	mean: (0.485, 0.456, 0.406)
	std: (0.229, 0.224, 0.225)
	crop_pct: 0.875
Test: [   0/196]  Time: 3.975s (3.975s,   64.39/s)  Loss:  9.3529 (9.3529)  Acc@1:   0.000 (  0.000)  Acc@5:   0.000 (  0.000)
Test: [  10/196]  Time: 0.258s (1.113s,  230.09/s)  Loss:  9.4012 (9.4141)  Acc@1:   0.000 (  0.036)  Acc@5:   0.000 (  0.178)
Test: [  20/196]  Time: 0.259s (1.067s,  239.82/s)  Loss:  9.4105 (9.4391)  Acc@1:   0.000 (  0.019)  Acc@5:   0.781 (  0.353)
Test: [  30/196]  Time: 0.257s (0.970s,  263.90/s)  Loss:  9.6106 (9.4592)  Acc@1:   0.000 (  0.038)  Acc@5:   0.391 (  0.340)
Test: [  40/196]  Time: 0.256s (0.994s,  257.67/s)  Loss:  9.4798 (9.4646)  Acc@1:   0.000 (  0.076)  Acc@5:   0.000 (  0.343)
Test: [  50/196]  Time: 0.257s (0.951s,  269.19/s)  Loss:  9.4490 (9.4665)  Acc@1:   0.000 (  0.084)  Acc@5:   0.391 (  0.345)
Test: [  60/196]  Time: 0.259s (0.969s,  264.10/s)  Loss:  9.4032 (9.4689)  Acc@1:   0.000 (  0.077)  Acc@5:   0.391 (  0.320)
Test: [  70/196]  Time: 0.258s (0.947s,  270.37/s)  Loss:  9.4011 (9.4626)  Acc@1:   0.391 (  0.094)  Acc@5:   1.172 (  0.325)
Test: [  80/196]  Time: 0.262s (0.958s,  267.17/s)  Loss:  9.5970 (9.4586)  Acc@1:   0.000 (  0.087)  Acc@5:   0.391 (  0.328)
Test: [  90/196]  Time: 0.260s (0.938s,  273.01/s)  Loss:  9.4456 (9.4596)  Acc@1:   0.000 (  0.082)  Acc@5:   0.391 (  0.326)
Test: [ 100/196]  Time: 0.259s (0.944s,  271.04/s)  Loss:  9.3502 (9.4556)  Acc@1:   0.000 (  0.093)  Acc@5:   0.391 (  0.333)
Test: [ 110/196]  Time: 0.258s (0.929s,  275.71/s)  Loss:  9.5069 (9.4557)  Acc@1:   0.000 (  0.095)  Acc@5:   0.391 (  0.334)
Test: [ 120/196]  Time: 0.262s (0.941s,  271.91/s)  Loss:  9.2863 (9.4555)  Acc@1:   0.391 (  0.094)  Acc@5:   0.391 (  0.333)
Test: [ 130/196]  Time: 0.259s (0.929s,  275.58/s)  Loss:  9.5734 (9.4552)  Acc@1:   0.000 (  0.092)  Acc@5:   0.000 (  0.334)
Test: [ 140/196]  Time: 0.262s (0.951s,  269.14/s)  Loss:  9.4510 (9.4544)  Acc@1:   0.000 (  0.089)  Acc@5:   1.172 (  0.335)
Test: [ 150/196]  Time: 0.260s (0.940s,  272.29/s)  Loss:  9.4878 (9.4513)  Acc@1:   0.000 (  0.091)  Acc@5:   0.781 (  0.344)
Test: [ 160/196]  Time: 0.260s (0.945s,  270.82/s)  Loss:  9.4090 (9.4509)  Acc@1:   0.000 (  0.092)  Acc@5:   0.000 (  0.340)
Test: [ 170/196]  Time: 0.262s (0.934s,  274.13/s)  Loss:  9.2623 (9.4503)  Acc@1:   0.000 (  0.089)  Acc@5:   0.391 (  0.338)
Test: [ 180/196]  Time: 0.262s (0.941s,  272.14/s)  Loss:  9.3247 (9.4489)  Acc@1:   0.391 (  0.093)  Acc@5:   0.781 (  0.352)
Test: [ 190/196]  Time: 1.471s (0.934s,  273.95/s)  Loss:  9.5550 (9.4490)  Acc@1:   0.000 (  0.092)  Acc@5:   0.781 (  0.360)
 * Acc@1 0.092 (99.908) Acc@5 0.360 (99.640)

---

2. On Windows (Pytorch 1.7.0+cu110):

python validate.py F:\ImageNet\newval2\ILSVRC2012_img_val\labelled --model resnet50 --pretrained

Result:

C:\Intel\pytorch_classifier>python validate.py F:\ImageNet\newval2\ILSVRC2012_im
g_val\labelled --model resnet50 --pretrained
C:\Users\Alex\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\_di
stributor_init.py:32: UserWarning: loaded more than 1 DLL from .libs:
C:\Users\Alex\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\.li
bs\libopenblas.NOIJJG62EMASZI6NYURL6JBKM4EVBGM7.gfortran-win_amd64.dll
C:\Users\Alex\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\.li
bs\libopenblas.PYQHXLVVQ7VESDPUVUADXEVJOBGHJPAY.gfortran-win_amd64.dll
  stacklevel=1)
Model resnet50 created, param count: 25557032
Data processing configuration for current model + dataset:
        input_size: (3, 224, 224)
        interpolation: bicubic
        mean: (0.485, 0.456, 0.406)
        std: (0.229, 0.224, 0.225)
        crop_pct: 0.875
C:\Users\Alex\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\_di
stributor_init.py:32: UserWarning: loaded more than 1 DLL from .libs:
C:\Users\Alex\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\.li
bs\libopenblas.NOIJJG62EMASZI6NYURL6JBKM4EVBGM7.gfortran-win_amd64.dll
C:\Users\Alex\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\.li
bs\libopenblas.PYQHXLVVQ7VESDPUVUADXEVJOBGHJPAY.gfortran-win_amd64.dll
  stacklevel=1)
C:\Users\Alex\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\_di
stributor_init.py:32: UserWarning: loaded more than 1 DLL from .libs:
C:\Users\Alex\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\.li
bs\libopenblas.NOIJJG62EMASZI6NYURL6JBKM4EVBGM7.gfortran-win_amd64.dll
C:\Users\Alex\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\.li
bs\libopenblas.PYQHXLVVQ7VESDPUVUADXEVJOBGHJPAY.gfortran-win_amd64.dll
  stacklevel=1)
C:\Users\Alex\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\_di
stributor_init.py:32: UserWarning: loaded more than 1 DLL from .libs:
C:\Users\Alex\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\.li
bs\libopenblas.NOIJJG62EMASZI6NYURL6JBKM4EVBGM7.gfortran-win_amd64.dll
C:\Users\Alex\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\.li
bs\libopenblas.PYQHXLVVQ7VESDPUVUADXEVJOBGHJPAY.gfortran-win_amd64.dll
  stacklevel=1)
C:\Users\Alex\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\_di
stributor_init.py:32: UserWarning: loaded more than 1 DLL from .libs:
C:\Users\Alex\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\.li
bs\libopenblas.NOIJJG62EMASZI6NYURL6JBKM4EVBGM7.gfortran-win_amd64.dll
C:\Users\Alex\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\.li
bs\libopenblas.PYQHXLVVQ7VESDPUVUADXEVJOBGHJPAY.gfortran-win_amd64.dll
  stacklevel=1)
Test: [   0/196]  Time: 5.817s (5.817s,   44.01/s)  Loss:  9.3646 (9.3646)  Acc@1:   0.000 (  0.000)  Acc@5:   0.000 (  0.000)
Test: [  10/196]  Time: 0.190s (0.871s,  293.93/s)  Loss:  9.4154 (9.4198)  Acc@1:   0.000 (  0.071)  Acc@5:   0.000 (  0.178)
Test: [  20/196]  Time: 0.189s (0.677s,  377.98/s)  Loss:  9.4082 (9.4416)  Acc@1:   0.000 (  0.037)  Acc@5:   0.781 (  0.372)
Test: [  30/196]  Time: 0.195s (0.585s,  437.80/s)  Loss:  9.6097 (9.4599)  Acc@1:   0.000 (  0.050)  Acc@5:   0.391 (  0.353)
Test: [  40/196]  Time: 0.189s (0.561s,  456.71/s)  Loss:  9.4744 (9.4647)  Acc@1:   0.000 (  0.086)  Acc@5:   0.000 (  0.353)
Test: [  50/196]  Time: 0.188s (0.523s,  489.44/s)  Loss:  9.4585 (9.4661)  Acc@1:   0.000 (  0.092)  Acc@5:   0.391 (  0.360)
Test: [  60/196]  Time: 0.188s (0.521s,  491.77/s)  Loss:  9.3945 (9.4687)  Acc@1:   0.000 (  0.083)  Acc@5:   0.391 (  0.339)
Test: [  70/196]  Time: 0.190s (0.501s,  510.93/s)  Loss:  9.4054 (9.4626)  Acc@1:   0.391 (  0.099)  Acc@5:   0.781 (  0.336)
Test: [  80/196]  Time: 0.189s (0.494s,  518.67/s)  Loss:  9.6091 (9.4591)  Acc@1:   0.000 (  0.092)  Acc@5:   0.391 (  0.338)
Test: [  90/196]  Time: 0.193s (0.482s,  530.67/s)  Loss:  9.4323 (9.4595)  Acc@1:   0.000 (  0.086)  Acc@5:   0.391 (  0.335)
Test: [ 100/196]  Time: 0.189s (0.480s,  533.42/s)  Loss:  9.3505 (9.4551)  Acc@1:   0.000 (  0.097)  Acc@5:   0.000 (  0.333)
Test: [ 110/196]  Time: 0.187s (0.469s,  545.99/s)  Loss:  9.5013 (9.4555)  Acc@1:   0.000 (  0.099)  Acc@5:   0.391 (  0.334)
Test: [ 120/196]  Time: 0.190s (0.470s,  544.86/s)  Loss:  9.2684 (9.4551)  Acc@1:   0.391 (  0.097)  Acc@5:   0.391 (  0.333)
Test: [ 130/196]  Time: 0.190s (0.462s,  554.33/s)  Loss:  9.5722 (9.4543)  Acc@1:   0.000 (  0.095)  Acc@5:   0.000 (  0.337)
Test: [ 140/196]  Time: 0.189s (0.460s,  556.65/s)  Loss:  9.4401 (9.4538)  Acc@1:   0.000 (  0.091)  Acc@5:   0.781 (  0.335)
Test: [ 150/196]  Time: 0.190s (0.454s,  563.40/s)  Loss:  9.4993 (9.4508)  Acc@1:   0.000 (  0.096)  Acc@5:   0.781 (  0.344)
Test: [ 160/196]  Time: 0.190s (0.453s,  565.14/s)  Loss:  9.4162 (9.4503)  Acc@1:   0.000 (  0.097)  Acc@5:   0.000 (  0.340)
Test: [ 170/196]  Time: 0.402s (0.448s,  571.46/s)  Loss:  9.2686 (9.4498)  Acc@1:   0.000 (  0.094)  Acc@5:   0.391 (  0.338)
Test: [ 180/196]  Time: 0.188s (0.448s,  571.57/s)  Loss:  9.3248 (9.4486)  Acc@1:   0.391 (  0.097)  Acc@5:   0.781 (  0.350)
Test: [ 190/196]  Time: 0.616s (0.446s,  573.90/s)  Loss:  9.5638 (9.4488)  Acc@1:   0.000 (  0.096)  Acc@5:   0.781 (  0.358)
 * Acc@1 0.096 (99.904) Acc@5 0.360 (99.640)

[rwightman]

Did you move validation images into folders? They are flat by default out of the tar. My dataset is folder based but you can always use the torchvision one with a one line change

https://www.github.com/tensorflow/models/tree/master/research%2Fslim%2Fdatasets%2Fpreprocess_imagenet_validation_data.py

@AlexeyAB this is one of the most tested scripts and models in the repo and I constantly run regression tests with it. Most likely the dataset layout

[AlexeyAB]

@rwightman Thanks!
Yes, it works. After applying the script:
python preprocess_imagenet_validation_data.py validation imagenet_2012_validation_synset_labels.txt

• https://github.com/tensorflow/models/blob/master/research/slim/datasets/preprocess_imagenet_validation_data.py
• https://github.com/tensorflow/models/blob/master/research/slim/datasets/imagenet_2012_validation_synset_labels.txt

I seem to have lost sight of this. It might be worth adding it here that ILSVRC2012_img_val dataset should be folder based: https://rwightman.github.io/pytorch-image-models/scripts/