How to save and evaluate model after training ?

[dnaveenr]

I wanted to know how we could save the model weights and use it for inference on some test data ?
I tried using torch.save() and torch.load() after all completion of all epochs, but the results are not similar as to automatic eval on test split and quite bad.
Could you please provide some inputs on this ?

[Confusezius]

You would need to show me exactly what you do. Ideally, you do:

1. opt = pkl.load(open(your_save_path/hypa.pkl','rb'))
2. network = netlib.networkselect(opt)
3. network.load_state_dict(torch.load(your_save_path/checkpoint.pth.tar)['state_dict'])
4. _ = network.eval()
5. with torch.no_grad(): do stuff

Also, you might not want to use the weights directly AFTER training, but those that provide the best validation performance (as is done automatically when calling eval.evaluate()). The reported results are based on weights found this way.

[dnaveenr]

Thanks @Confusezius ; will test with your inputs. I was saving the model.state_dict after every epoch and picked the model file with best validation performance.
I am working on image retrieval with L2 distance based on the feature embeddings generated after training. I noticed that directly generating the feature vector embedding ( with Resnet pretrained model ) gives results which were specific to image color features, so currently working on improving the embeddings.

[Confusezius]

@dnaveenr keep me posted, both if everything works out or not :)

[dnaveenr]

@Confusezius Sure. Will keep you posted.
Just wanted to check more one thing, since we are doing online sampling, does batch size play a major role in performance ? Currently, default batch size is 112, which I cannot use on my GPU. I'm working with a batch size of 32. During your tests, did you try with different batch size ? Any suggestions ?

[Confusezius]

So for methods like ProxyNCA where you use external proxies to create meaningful n-tuples for training it doesn't matter (to a certain degree). However, for triplet sampling methods like semihard or distance sampling, there is a notable drop in performance for small batchsizes, as the set of triplets you can create per choice of anchor gets smaller, thereby constraining the effectiveness of your sampling method. In that regard, I have seen significant performance change for batchsizes below 100. However you can try and counter it a bit by maybe making --n_samples_per_class smaller, e.g. using 2 or 3 instead of 4, to uphold a higher diversity in negatives.

[dnaveenr]

Thanks. Very insightful. I will try tweaking --n_samples_per_class and check.

[Confusezius]

Feel free to reopen this issue if there are any other issues :). Otherwise I'll close it for now.