Low Test Accuracy using Inception_V3

[soumendukrg]

I found out that the test accuracy for pretrained Inception_V3 model on ImageNet dataset in distiller are not the same as reported by the paper as well as by torchvision(PyTorch) documentation. Here are the results:
Top 1: 69.538 Top5: 88.654
Expected Top 1: 77.45 Top5: 93.56

I am using the following code to evaluate:
$ python compress_classifier.py /path_to_imagenet2012/ -a=inception_v3 --gpu 0 -e --pretrained

However, the accuracy for other networks like ResNet, DenseNet, AlexNet seemed to be almost similar to the ones reported in PyTorch docs. Do I have to specify any additional argument for Inception_V3?

[soumendukrg]

I found out that the file data_loaders has fixed classification image size for imagenet as 1, 3, 224, 224. However, PyTorch inception_v3 requires image size of 1, 3, 299, 299.

When I modified the file to use the image size corresponding to inception, the test accuracy reported are:
Top1: 77.318 Top5: 93.402 , which are almost same as reported by Torchvision classification models.

This file and linked files needs to be modified. @nzmora unless there is any other method to fix this issue.

[nzmora]

Hi @soumendukrg,

Thanks - this is an important bug to fix!
Would you care to issue a PR with the fix? It would help speed the delivery of a fix to others.
Thanks!
Neta

[soumendukrg]

Thanks for responding. I will submit a PR with the fix. In fact for training we need to change data size as well as loss function in image_classifier.py, at line 501.

    if not early_exit_mode(args):
        if isinstance(output, tuple):
            loss = sum((criterion(o,target) for o in output))
        else:
            loss = criterion(output, target)

        # Measure accuracy
        if isinstance(output, tuple):
            classerr.add(output[0].data, target)
        else:
            classerr.add(output.data, target)`

This adds the aux logits loss with normal loss thereby increasing training efficiency, but I read somewhere that we can omit that loss since during validation/test, that part is essentially cut of from network. In that case, the above code will change. What do you suggest I do for this part?

[nzmora]

Thanks @soumendukrg for the fix! It took some time, but we finally merged the PR.

[soumendukrg]

Thanks @nzmora for the merge. I have been working on the object detection compression and have one suggestion/fix which I will put up in a separate issue.