ValueError: Cannot feed value of shape (1, 3, 299, 299) for Tensor 'input_1_1:0', which has shape '(?, ?, ?, 3)' while using visualize cam

[Abhijit-2592]

hey @raghakot, @sakares, I am trying to use visualize cam on my network (a fine-tuned inceptionv3)
here is my code

# keras 
from keras import backend as K
from keras import activations
from keras.models import model_from_json
from keras.applications.inception_v3 import InceptionV3
# keras vis
from vis.utils import utils
from vis.visualization import visualize_cam,saliency,overlay
# image processing
import cv2
import scipy.misc
# others
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
# default backend is tensorflow

K.set_image_data_format('channels_last')
K.set_image_dim_ordering('tf')
model_path = '/workspace/abhijit/project_dent/models/M16_SIM21_inv3_PT_dents_v2_flavour4_ACC_89,90.json'
weights_path = '/workspace/abhijit/project_dent/weights/BW16_SIM21_inv3_PT_dents_v2_flavour4_ACC_89,90.hdf5'
image = '/workspace/abhijit/project_dent/miscellaneous_codes/images/HUB693_OP20_C1_P_0001_Pixel_257_513.jpg'

# read the image
img = scipy.misc.imread(image,mode='RGB')
img = scipy.misc.imresize(img,(299,299))
print(img.shape)

plt.imshow(img)

float_img = img.astype('float32')
plt.imshow(float_img)
plt.show()
print(float_img.shape)

%%time
# load the model and weights 
json_file = open(model_path, 'r')
loaded_model_json = json_file.read()
json_file.close()
loaded_model = model_from_json(loaded_model_json)
# load weights into new model
loaded_model.load_weights(weights_path)
print("Loaded model from disk")

model = loaded_model

# Utility to search for layer index by name
layer_idx = utils.find_layer_idx(model,'dense_1')
print(layer_idx)

#swap softmax with linear classifier 
model.layers[layer_idx].activation = activations.linear
model = utils.apply_modifications(model)

penultimate_layer_idx = utils.find_layer_idx(model,'mixed10')
print(penultimate_layer_idx)

%%time
heatmap = visualize_cam(model, 
                        layer_idx=layer_idx, 
                        filter_indices=[0], #for with_dent
                        seed_input=float_img, 
                        penultimate_layer_idx = penultimate_layer_idx,
                        backprop_modifier = None
                        )

when i run this i am getting a weird value error:

**
ValueError Traceback (most recent call last)
in ()

/opt/conda/lib/python3.5/site-packages/vis/visualization/saliency.py in visualize_cam(model, layer_idx, filter_indices, seed_input, penultimate_layer_idx, backprop_modifier, grad_modifier)
237 (ActivationMaximization(model.layers[layer_idx], filter_indices), -1)
238 ]
--> 239 return visualize_cam_with_losses(model.input, losses, seed_input, penultimate_layer, grad_modifier)

/opt/conda/lib/python3.5/site-packages/vis/visualization/saliency.py in visualize_cam_with_losses(input_tensor, losses, seed_input, penultimate_layer, grad_modifier)
158 penultimate_output = penultimate_layer.output
159 opt = Optimizer(input_tensor, losses, wrt_tensor=penultimate_output, norm_grads=False)
--> 160 _, grads, penultimate_output_value = opt.minimize(seed_input, max_iter=1, grad_modifier=grad_modifier, verbose=False)
161
162 # For numerical stability. Very small grad values along with small penultimate_output_value can cause

/opt/conda/lib/python3.5/site-packages/vis/optimizer.py in minimize(self, seed_input, max_iter, input_modifiers, grad_modifier, callbacks, verbose)
141
142 # 0 learning phase for 'test'
--> 143 computed_values = self.compute_fn([seed_input, 0])
144 losses = computed_values[:len(self.loss_names)]
145 named_losses = zip(self.loss_names, losses)

/src/keras/backend/tensorflow_backend.py in call(self, inputs)
2266 updated = session.run(self.outputs + [self.updates_op],
2267 feed_dict=feed_dict,
-> 2268 **self.session_kwargs)
2269 return updated[:len(self.outputs)]
2270

/opt/conda/lib/python3.5/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
787 try:
788 result = self._run(None, fetches, feed_dict, options_ptr,
--> 789 run_metadata_ptr)
790 if run_metadata:
791 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/opt/conda/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
973 'Cannot feed value of shape %r for Tensor %r, '
974 'which has shape %r'
--> 975 % (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
976 if not self.graph.is_feedable(subfeed_t):
977 raise ValueError('Tensor %s may not be fed.' % subfeed_t)

ValueError: Cannot feed value of shape (1, 3, 299, 299) for Tensor 'input_1_1:0', which has shape '(?, ?, ?, 3)'
**

My model definition code is given below:

#create a base model
base_model = InceptionV3(weights='imagenet',include_top=False)
# add a global average pooling layer
x = base_model.output
x = GlobalAveragePooling2D(name='avg_pool')(x)
# output classifier with 2 outputs
predictions = Dense(2,activation='softmax')(x)
#combine input and output and create a model
model = Model(inputs=base_model.input,outputs=predictions)
# unfreeze the model.
for layer in base_model.layers:
    layer.trainable = True
# compile the model
sgd = optimizers.SGD(lr=0.001 , momentum = 0.0, decay=0.0 ,nesterov=True)
model.compile(loss ='categorical_crossentropy',optimizer=sgd, metrics=['accuracy'])

I am clueless why it is showing this value error.

The error looks like a clash between tensorflow and theano background. Idk why this is happening because I have made the keras config file to run tensorflow by default.
could you please help me sort this issue out?
Thanks in advance

**update :
1)This error doesn't come up when I load the full inceptionv3 model from keras (including the top layer as given in #65
2) same error with visualize_saliency also **

[sakares]

You might need to set the input_shape parameter when assigning InceptionV3 with the proper input shape. This is my usual setting:

if K.image_data_format() == 'channels_first':
   input_shape = (3, img_width, img_height)
else:
   input_shape = (img_width, img_height, 3)

base_model = InceptionV3(input_shape=input_shape, weights='imagenet', include_top=False)

Not sure whenever you have set K.set_image_data_format('channels_last') and
K.set_image_dim_ordering('tf') could let the model initialization works properly.

Hope it helps

[Abhijit-2592]

@sakares thanks a ton! its working!!!!!

I will close this issue now

[dietercastel]

@sakares solution works indeed. But since I fine-tuned some models without specifying input_shape before and didn't want to retrain them all just for this I figured out a way to fix it afterwards:

Assume newModel() returns a new model where the input_shape is set as sakares explained. The structure of newModel() should be identical to the pertrained one, except for the input_shape.

import sys
from keras.models import Model,load_model
modelArg = sys.argv[1] #read filename from command line
oldmodel=load_model(modelArg)

model = newModel()

#Some sanity checking with assertions.
assert len(oldmodel.layers) == len(model.layers)
for idx,layer in enumerate(oldmodel.layers):
    assert type(oldmodel.layers[idx]) == type(model.layers[idx])
    model.layers[idx].set_weights(layer.get_weights())

#Sometimes keras will complain if the model isn't compiled already so compile at will
model.compile(optimizer=SGD(lr=0.001, momentum=0.9), loss='categorical_crossentropy')
#save it again or use it with keras-vis

It takes a bit of time to transfer all the weights but orders of magnitude less than retraining the whole network.