Identical & low confidence predictions

[several27]

Please make sure that the boxes below are checked before you submit your issue.

Thank you!

• Check that you are up-to-date with the master branch of Keras. You can update with:

  pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps
  

• If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

• If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:

  pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps
  

• Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

Hi, first of all, huge thanks for creating this repo and training the models!

I'm running into a weird problem, where I can't seem to run even the most simple example:

from vgg16_places_365 import VGG16_Places365
from keras.preprocessing import image

model = VGG16_Places365(weights='places')

img_path = 'restaurant.jpg'
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

preds = model.predict(x)
print('Predicted:', preds)

The code above is missing the preprocess_input, so I tried with keras, pytorch and my own implementations of it (by guessing what it does). But I keep getting basically the same predictions, doesn't matter on what image I run them on or what preprocess function I use (all from places365 dataset):

0.023 -> museum/indoor
0.023 -> coffee_shop
0.022 -> art_studio
0.021 -> campus
0.019 -> yard
0.019 -> inn/outdoor
0.016 -> science_museum
0.015 -> motel
0.015 -> building_facade
0.015 -> staircase

Not sure whether that's a problem with my preprocess_input method or something else, but I'd really appreciate some help!

[GKalliatakis]

Hi, this is the official def preprocess_input. However, I don't thing your problem lies there, since def preprocess_input only preprocesses tensor to be compatible with images coming from caffe-like format or tf-like format.

If you have a look at the official documentation HERE

print('Predicted:', decode_predictions(preds, top=3)[0])
# Predicted: [(u'n02504013', u'Indian_elephant', 0.82658225), (u'n01871265', u'tusker', 0.1122357), (u'n02504458', u'African_elephant', 0.061040461)]

You'll need to have a function similar to def decode_predictions, which is not included in the provided code. If you are interested we can make it work together by writing a similar function based on the the category index file from the official repo of Places and add it to this repo.

[GKalliatakis]

@several27 Can you please try now with the newly created def decode_predictions from places_utils.py? Also you can have a look at places365_class_index.json. Please comment below with your results. Hope it helps.

[Oktai15]

@GKalliatakis anyway, you should add from places_utils import preprocess_input in your examples in readme

[GKalliatakis]

@Oktai15 Thanks for that. Examples have now been updated.

[ppgiannak]

I can confirm i have the same problem. No matter the image i get very small confidence scores.
I think in your code, the url does not download the correct h5 file.

[GKalliatakis]

@ppgiannak You are right. I have dumped out the weights from scratch again but I keep getting the same predicted classes with low (0.023) confidence scores no matter what the selected image is. Although feature extraction seems to work fine. Would you like to work on that? The JSON class_index file has now been added to releases

[ppgiannak]

I have already tried several tools available to transfer the pre-trained weights from Caffe to Keras but with no luck. I may have to develop my own.
The tricky part lies in Keras configuration. It all depends on what back-end you are running and the dimension ordering you are using (channels_last or channels_first). I have made some progress for a Theano backend but i want to get it working for Tensorflow ultimately.

[GKalliatakis]

There is no dedicated tool for converting Caffe weights to Keras models, that's why I have created Keras Application Zoo - A public clearinghouse for Keras Applications-like image classification models to promote progress and reproduce research. The original model structure must be manually defined and then transfer the converted weights layer by layer (this is what I've done for this repo). I don't believe dimension ordering and the type of back-end poses the greatest threat for that. I have successfully tested Places for extracting features, but I had never used it for directly for classifying images with the defined categories. Any help will be appreciated.

[ramprs]

I am facing the same problem (identical non-confident predictions) with the given example code. Were anyone able to figure this out yet? Thanks

[DongChen06]

I am experiencing the same problem (low confident predictions) with the given example code. Were anyone able to figure this out yet? Thank you very much!

[DongChen06]

I think the error may come from the wrong pre-trained weights

[GKalliatakis]

I have tried dumping out the original caffe weights again from scratch (for both Places365 [which is the latest version] and Places205 [previous version]) but there seems to be the same error, which leads me to believe that there is an error with my code for dumping out caffe weights.

The code can be found here

Can anyone help with exporting the original weights from caffe to numpy arrays?

[atunick]

a) I suspect that the test script for the VGG16 Places 365 model is hardwired for the jpg image 'restaurant.jpg' as shown in the example on the github. Is there a way to change a configuration file and/or a part of a specific .py code to allow for new and different test images?
*** It appears from the discussion above that this has to do with the imported weights for the pre-trained ImageNet+Places CNN. ***
For example, for the image 'restaurant.jpg' the top-5 classes are [99, 236, 21, 77, 193] corresponding to the top-5 categories coffee_shop, museum/indoor, art_studio, campus, inn/outdoor.
For the image 'tank_forest.jpg', similar classes are calculated, i.e., [99 236 21 77 362].
.
b) When I modified the VGG places 365 script to test the VGG16_hybrid_places_1365, I expected the "preds" variable to have shape (1365L) and the 'top_preds' variable to have shape (5L). Instead they are (7L, 7L, 512L) and (5, 7L, 512L), respectively.
.
*** changing the line of code where include_top=False to include_top=True fixes the problem. Now, the top-10 results for 'restaurant.jpg' are:
preds: dim=1; shape=(1365L,)
top-10 confidence levels= [ 0.02964255 0.02542448 0.01824417 0.01738015 0.01657032 0.01656142
0.01593293 0.01589166 0.01564833 0.01499115]
top-10 classes = [1236 1099 1067 1231 1220 1259 1077 1193 1185 1297]
top-10 categories: museum/indoor; coffee_shop; building_facade; motel; mansion; patio; campus; inn/outdoor; ice_cream_parlor; science_museum.
.
*** Please let me know when the pre-trained hybrid 1365 CNN weights are corrected/updated, Perhaps then some of the ImageNet 0-999 classes will be included in the results.***

[atunick]

test model code for VGG16_hybrid_places_1365:
==============================.
import keras
import numpy as np
import os
from VGG16_hybrid_places_1365 import VGG16_Hubrid_1365
from keras.preprocessing import image
from places_utils import preprocess_input
model = VGG16_Hubrid_1365(weights='places', include_top=True)
#model = VGG16_Hubrid_1365(weights='places', include_top=False)
img_path = 'restaurant.jpg'
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)
print x.ndim, x.shape
predictions_to_return = 10
preds = model.predict(x)[0]
print 'preds', preds.ndim, preds.shape, np.sort(preds)[::-1][0:10]
top_preds = np.argsort(preds)[::-1][0:10]
print'top_preds', top_preds.ndim, top_preds.shape, top_preds
print '  '
# load the class label
file_name = 'categories_hybrid1365.txt'
if not os.access(file_name, os.W_OK):
    synset_url = 'https://raw.githubusercontent.com/CSAILVision/places365/master/categories_hybrid1365.txt'
    os.system('wget ' + synset_url)
classes = list()
with open(file_name) as class_file:
    for line in class_file:
        classes.append(line.strip().split(' ')[0][3:])
classes = tuple(classes)
print('--SCENE CATEGORIES:')
# output the prediction
for i in range(0, 10):
    print(classes[top_preds[i]])

[ppgiannak]

Ok, it looks like there is a lot of interest in those.
Unfortunately i haven't made any useful progress but i think the test script is not relevant to the weight conversion. I am not able to work on that right now, but maybe in the future. I'll post here in case i have any news.

P.S.: Please don't email me directly, keeping the conversation open in github is the best practice always. :)

[atunick]

@ppgiannak Thank you. Looking forward.

[kelly-xuxixi]

I faced the same problem when using the provided weights to make predictions. Maybe you can add a WARNING in Readme.md to save others' time. Thanks for your effort anyway.

[pavelgonchar]

You can use our keras versions of VGG16Places365 and VGG16PlacesHybrid1365 that were converted from original caffe files

https://github.com/antorsae/landmark-recognition-challenge/blob/master/extra/vgg16_places365.py

Here is a demo:

https://gist.github.com/pavelgonchar/4ebdb19b575eb3b102ad0840563b14c3

[GKalliatakis]

Thanks for that @pavelgonchar.
Can you please elaborate more on how did you manage to convert the original caffe weights?

[GKalliatakis]

This issue is now fixed thanks to the @pavelgonchar and his release with the correct converted weights found in landmark-recognition-challenge.

Also the README file has been updated accordingly.

This issue will now close.