Getting model activations with K.function doesn't work since v2.2.0

[ViaFerrata]

Hi there,

I'm running into problems with getting the activations of a model using K.function since Keras 2.2.0. With 2.1.6, everything works fine. I'm using the tensorflow backend with a convnet with multiple outputs.

A (shortened) code snippet can be found below:

saved_model = ks.models.load_model('...') # the model has been saved with 2.2.0
inp = saved_model.input

generator = generate_batches_from_hdf5_file(...)
model_inputs, ys = next(generator) 
lp = 0. if learning_phase == 'test' else 1.

outputs = [layer.output for layer in saved_model.layers if
               layer.name == layer_name or layer_name is None]  # all layer outputs -> empty tf.tensors 

funcs = [K.function([inp] + [K.learning_phase()], [out]) for out in outputs]  # evaluation functions

list_inputs = [model_inputs, lp]

layer_outputs = [func(list_inputs)[0] for func in funcs] <- throws an error since Keras 2.2.0
...

The last line yields the following error since 2.2.0:

File ".../visualization_tools.py", line 135, in get_activations_and_weights
    layer_outputs = [func(list_inputs)[0] for func in funcs]
  File "/home/hpc/capn/mppi033h/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 2661, in __call__
    return self._call(inputs)
  File "/home/hpc/capn/mppi033h/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 2630, in _call
    session)
  File "/home/hpc/capn/mppi033h/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 2582, in _make_callable
    callable_fn = session._make_callable_from_options(callable_opts)
  File "/home/hpc/capn/mppi033h/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1480, in _make_callable_from_options
    return BaseSession._Callable(self, callable_options)
  File "/home/hpc/capn/mppi033h/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1441, in __init__
    session._session, options_ptr, status)
  File "/home/hpc/capn/mppi033h/.local/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py", line 519, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: input_1_1:0 is both fed and fetched.
Exception tensorflow.python.framework.errors_impl.InvalidArgumentError: InvalidArgumentError() in <bound method _Callable.__del__ of <tensorflow.python.client.session._Callable object at 0x7f35b1f20a10>> ignored

Any idea what could have changed in 2.2.0?

[fchollet]

Please provide a standalone error reproduction code snippet (something that can be copy/pasted and run as-in, producing this error).

[ViaFerrata]

Here's a standalone code that reproduces the error with Keras 2.2.0 and works with v2.1.6:

from keras.layers import Input, Dense, Activation, Flatten, Convolution3D, BatchNormalization
from keras import backend as K
import numpy as np

def create_cnn(nb_classes):
    """
    Returns a cnn model.
    """
    input_layer = Input(shape=(11,13,18,60), dtype=K.floatx())  # input_layer

    x = Convolution3D(64, (3,) * 3, use_bias=False)(input_layer)

    x = BatchNormalization(axis=-1)(x)
    x = Activation('relu')(x)

    x = Flatten()(x)
    x = Dense(256)(x)
    x = Activation('relu')(x)

    label_names = ['neuron_' + str(i) for i in range(nb_classes)]
    x = [Dense(1, name=name)(x) for name in label_names]

    model = Model(inputs=input_layer, outputs=x)

    return model

batchsize = 32
model_input = np.random.randint(0, 255, (batchsize,11,13,18,60))

nn_model = create_cnn(5)
inp = nn_model.input
inp = [inp]  # only one input! let's wrap it in a list.
lp = 0.

layer_name = None
outputs = [layer.output for layer in nn_model.layers if
           layer.name == layer_name or layer_name is None]  # all layer outputs -> empty tf.tensors

funcs = [K.function(inp + [K.learning_phase()], [out]) for out in outputs]  # evaluation functions

list_inputs = [model_input, lp]
layer_outputs = [func(list_inputs)[0] for func in funcs]

The error:

Traceback (most recent call last):
  File "keras_error.py", line 44, in <module>
    layer_outputs = [func(list_inputs)[0] for func in funcs]
  File "keras_error.py", line 44, in <listcomp>
    layer_outputs = [func(list_inputs)[0] for func in funcs]
  File "/home/hpc/capn/mppi013h/python_conda_environment/neuralnets/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2661, in __call__
    return self._call(inputs)
  File "/home/hpc/capn/mppi013h/python_conda_environment/neuralnets/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2630, in _call
    session)
  File "/home/hpc/capn/mppi013h/python_conda_environment/neuralnets/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2582, in _make_callable
    callable_fn = session._make_callable_from_options(callable_opts)
  File "/home/hpc/capn/mppi013h/python_conda_environment/neuralnets/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1480, in _make_callable_from_options
    return BaseSession._Callable(self, callable_options)
  File "/home/hpc/capn/mppi013h/python_conda_environment/neuralnets/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1441, in __init__
    session._session, options_ptr, status)
  File "/home/hpc/capn/mppi013h/python_conda_environment/neuralnets/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 519, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: input_1:0 is both fed and fetched.
Exception ignored in: <bound method BaseSession._Callable.__del__ of <tensorflow.python.client.session.BaseSession._Callable object at 0x7f802deede48>>
Traceback (most recent call last):
  File "/home/hpc/capn/mppi013h/python_conda_environment/neuralnets/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1464, in __del__
    self._session._session, self._handle, status)
  File "/home/hpc/capn/mppi013h/python_conda_environment/neuralnets/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 519, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: No such callable handle: 337385904 

[sx5640]

Similar issue found while using the visualization tool keras-vis. It throws InvalidArgumentError: No such callable handle in 2.2.0 but not in 2.1.6.
Here is full traceback:

Traceback (most recent call last):
  File "visualize.py", line 112, in <module>
    ARGS.func(**vars(ARGS))
  File "visualize.py", line 67, in plt_attention
    grads = visualize_saliency(model, layer_idx, filter_indices=None, seed_input=img)
  File "D:\Code\mura\env\lib\site-packages\vis\visualization\saliency.py", line 125, in visualize_saliency
    return visualize_saliency_with_losses(model.input, losses, seed_input, grad_modifier)
  File "D:\Code\mura\env\lib\site-packages\vis\visualization\saliency.py", line 73, in visualize_saliency_with_losses
    grads = opt.minimize(seed_input=seed_input, max_iter=1, grad_modifier=grad_modifier, verbose=False)[1]
  File "D:\Code\mura\env\lib\site-packages\vis\optimizer.py", line 143, in minimize
    computed_values = self.compute_fn([seed_input, 0])
  File "D:\Code\mura\env\lib\site-packages\keras\backend\tensorflow_backend.py", line 2661, in __call__
    return self._call(inputs)
  File "D:\Code\mura\env\lib\site-packages\keras\backend\tensorflow_backend.py", line 2630, in _call
    session)
  File "D:\Code\mura\env\lib\site-packages\keras\backend\tensorflow_backend.py", line 2582, in _make_callable
    callable_fn = session._make_callable_from_options(callable_opts)
  File "D:\Code\mura\env\lib\site-packages\tensorflow\python\client\session.py", line 1480, in _make_callable_from_options
    return BaseSession._Callable(self, callable_options)
  File "D:\Code\mura\env\lib\site-packages\tensorflow\python\client\session.py", line 1441, in __init__
    session._session, options_ptr, status)
  File "D:\Code\mura\env\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 519, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: input_1_1:0 is both fed and fetched.
Exception ignored in: <bound method BaseSession._Callable.__del__ of <tensorflow.python.client.session.BaseSession._Callable object at 0x0000014EA4DAADA0>>
Traceback (most recent call last):
  File "D:\Code\mura\env\lib\site-packages\tensorflow\python\client\session.py", line 1464, in __del__
    self._session._session, self._handle, status)
  File "D:\Code\mura\env\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 519, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: No such callable handle: 1436883128496

[JennaElli]

Similar issue here too. I'm trying to run the example script for 'Activation Maximization on VGGNet'.

Traceback (most recent call last):
File "kvis.py", line 24, in
img = visualize_activation(model, layer_idx, filter_indices=20)
File "/home/ycc62267/.local/lib/python3.6/site-packages/vis/visualization/activation_maximization.py", line 107, in visualize_activation
return visualize_activation_with_losses(model.input, losses, seed_input, input_range, **optimizer_params)
File "/home/ycc62267/.local/lib/python3.6/site-packages/vis/visualization/activation_maximization.py", line 41, in visualize_activation_with_losses
img = opt.minimize(**optimizer_params)[0]
File "/home/ycc62267/.local/lib/python3.6/site-packages/vis/optimizer.py", line 143, in minimize
computed_values = self.compute_fn([seed_input, 0])
File "/dls_sw/apps/python/anaconda/1.7.0/64/envs/python3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2661, in call
return self._call(inputs)
File "/dls_sw/apps/python/anaconda/1.7.0/64/envs/python3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2630, in _call
session)
File "/dls_sw/apps/python/anaconda/1.7.0/64/envs/python3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2582, in _make_callable
callable_fn = session._make_callable_from_options(callable_opts)
File "/dls_sw/apps/python/anaconda/1.7.0/64/envs/python3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1480, in _make_callable_from_options
return BaseSession._Callable(self, callable_options)
File "/dls_sw/apps/python/anaconda/1.7.0/64/envs/python3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1441, in init
session._session, options_ptr, status)
File "/dls_sw/apps/python/anaconda/1.7.0/64/envs/python3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 519, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: input_1_1:0 is both fed and fetched.
Exception ignored in: <bound method BaseSession._Callable.del of <tensorflow.python.client.session.BaseSession._Callable object at 0x7f1417e97be0>>
Traceback (most recent call last):
File "/dls_sw/apps/python/anaconda/1.7.0/64/envs/python3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1464, in del
self._session._session, self._handle, status)
File "/dls_sw/apps/python/anaconda/1.7.0/64/envs/python3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 519, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: No such callable handle: 139726883220512

[yanboliang]

@ViaFerrata After checking your code, I found the input_layer was passed into both feed and fetch. This is illegal for TF since keras switch to use tf.Session._make_callable_from_options. You can insert outputs = outputs[1:] before

funcs = [K.function(inp + [K.learning_phase()], [out]) for out in outputs]

to remove the first input_layer from fetch, then it can work well.