Failed in fine-tuning inception_v3

[JingyunLiang]

I failed in using inception_v3 on my own dataset. (Ubuntu14.04, cuda8.0, python3.6.2)

It outputs warning when loaded:

/home/ljy/anaconda3/lib/python3.6/site-packages/torchvision-0.1.9-py3.6.egg/torchvision/models/inception.py:65: UserWarning: src is not broadcastable to dst, but they have the same number of elements.  Falling back to deprecated pointwise behavior.

It failed which training:

Traceback (most recent call last):
  File "/home/ljy/pytorch-examples-master/cub_pytorch/main.py", line 382, in <module>
    main()
  File "/home/ljy/pytorch-examples-master/cub_pytorch/main.py", line 213, in main
    train(train_loader, model, criterion, optimizer, epoch)
  File "/home/ljy/pytorch-examples-master/cub_pytorch/main.py", line 251, in train
    loss = criterion(output, target_var)
  File "/home/ljy/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ljy/anaconda3/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 482, in forward
    self.ignore_index)
  File "/home/ljy/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 746, in cross_entropy
    return nll_loss(log_softmax(input), target, weight, size_average, ignore_index)
  File "/home/ljy/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 537, in log_softmax
    return _functions.thnn.LogSoftmax.apply(input)
  File "/home/ljy/anaconda3/lib/python3.6/site-packages/torch/nn/_functions/thnn/auto.py", line 126, in forward
    ctx._backend = type2backend[type(input)]
  File "/home/ljy/anaconda3/lib/python3.6/site-packages/torch/_thnn/__init__.py", line 15, in __getitem__
    return self.backends[name].load()
KeyError: <class 'tuple'>

[alykhantejani]

Hi @MichaelLiang12,

What PyTorch version are you using (found by torch.__version__), also can you provide us with a minimum working example to reproduce this?

Thanks

[alykhantejani]

Also the user warning you are getting when loading the model is fixed in master (via #231)

[jamiechoi1995]

Same issue:

(tensorflow) wcai@tdtd-desktop ~/tensorflow/AI_competition/pytorch $ python main.py -a inception_v3 . --pretrained
=> using pre-trained model 'inception_v3'
/home/wcai/tensorflow/lib/python3.5/site-packages/torchvision/models/inception.py:65: UserWarning: src is not broadcastable to dst, but they have the same number of elements. Falling back to deprecated pointwise behavior.
m.weight.data.copy_(values)
Traceback (most recent call last):
File "main.py", line 353, in
main()
File "main.py", line 176, in main
train(train_loader, model, criterion, optimizer, epoch)
File "main.py", line 214, in train
loss = criterion(output, target_var)
File "/home/wcai/tensorflow/lib/python3.5/site-packages/torch/nn/modules/module.py", line 224, in call
result = self.forward(*input, **kwargs)
File "/home/wcai/tensorflow/lib/python3.5/site-packages/torch/nn/modules/loss.py", line 482, in forward
self.ignore_index)
File "/home/wcai/tensorflow/lib/python3.5/site-packages/torch/nn/functional.py", line 746, in cross_entropy
return nll_loss(log_softmax(input), target, weight, size_average, ignore_index)
File "/home/wcai/tensorflow/lib/python3.5/site-packages/torch/nn/functional.py", line 537, in log_softmax
return _functions.thnn.LogSoftmax.apply(input)
File "/home/wcai/tensorflow/lib/python3.5/site-packages/torch/nn/_functions/thnn/auto.py", line 126, in forward
ctx._backend = type2backend[type(input)]
File "/home/wcai/tensorflow/lib/python3.5/site-packages/torch/_thnn/init.py", line 15, in getitem
return self.backends[name].load()
KeyError: <class 'tuple'>

Python: Python 3.5.2

print (torch.version)
0.2.0_3

[alykhantejani]

Hi @jamiechoi1995,

Can you provide a minimum working example of this failing (i.e. an input that causes this when you pass it the model).

From the stack trace it seems like the input to the loss is a tuple, instead of a Variable.

[jamiechoi1995]

Hi @alykhantejani

You can reproduce this problom by using the code in https://github.com/pytorch/examples/tree/master/imagenet

I modify the size of rescale and crop to 299 for inception v3,
and my train&validate data are jpg files and the corresponding json files.

Using the same code with size of 224 in resnet model is OK,
but when I swith it to inception v3, I got this problem.

Thanks.

[TiRune]

Isn't this problem because the Aux error branch in the network? If you remove it it should work :)

[alykhantejani]

@jamiechoi1995 @MichaelLiang12, @TiRune is correct, inception_v3 has an aux branch, and if this is not disabled the forward function will return a tuple (see here), which when passed to the criterion will throw this error.

So you have two choices:

1. disable aux_logits when the model is created here by also passing aux_logits=False to the inception_v3 function.

2. edit your train function to accept and unpack the returned tuple here to be something like:

output, aux = model(input_var)

[rajasekharponakala]

@alykhantejani:Hi, why we have to disable the aux_logits?, what are these aux_logits? does they effect the training/validation?

I'm trying to reproduce the accuracy from a model trained using with the bvlc_googlenet (without pretrained weights). So when I do aux branch off with pytorch(googlenet) it works and reports val_acc with 50% which is very low when compared to the caffe. any other methods to reproduce the same accurcy using pytorch?
Thanks.

@jamiechoi1995 @MichaelLiang12, @TiRune is correct, inception_v3 has an aux branch, and if this is not disabled the forward function will return a tuple (see here), which when passed to the criterion will throw this error.

So you have two choices:

1. disable `aux_logits` when the model is created [here](https://github.com/pytorch/examples/blob/master/imagenet/main.py#L75) by also passing `aux_logits=False` to the `inception_v3` function.

2. edit your `train` function to accept and unpack the returned tuple [here](https://github.com/pytorch/examples/blob/master/imagenet/main.py#L194) to be something like:

output, aux = model(input_var)

[fmassa]

@rajasekharponakala the aux_logits is a separate classifier that is added to help during training, but it is not used during inference.

I'm trying to reproduce the accuracy from a model trained using with the bvlc_googlenet (without pretrained weights). So when I do aux branch off with pytorch(googlenet) it works and reports val_acc with 50% which is very low when compared to the caffe. any other methods to reproduce the same accurcy using pytorch?

Both googlenet and inception_v3 use pre-trained weights from TensorFlow, and as far as I know we didn't manage to reproduce accuracies from the paper when training from scratch.

[rajasekharponakala]

Hi @fmassa, thanks. I followed (pytorch discourse) to add below lines in train() imagenet example.

output, aux = model(input_var)
loss1 = criterion(output, target)
loss2 = criterion(aux, target)
loss = loss1 + 0.4*loss2

but ended with error:

Traceback (most recent call last):
  File "imagenet.py", line 407, in <module>
    main()
  File "imagenet.py", line 114, in main
    main_worker(args.gpu, ngpus_per_node, args)
  File "imagenet.py", line 240, in main_worker
    train(train_loader, model, criterion, optimizer, epoch, args)
  File "imagenet.py", line 281, in train
    output, aux = model(input)
ValueError: too many values to unpack (expected 2)

any idea?

[fmassa]

you need to set your model to train() mode, it's probably in eval mode