AttributeError _thread._local error on inference when using Keras + Tensorflow

[RobertLucian]

Version

On version 0.13.x.

Description

When using the latest version of Keras (2.3.1) and Tensorflow as its backend (2.x), the AttributeError: '_thread._local' object has no attribute 'value' error is encountered because _SYMBOLIC_SCOPE.value does not exist within the Tensorflow backend. The error is encountered when making the prediction and not when the model is loaded.

It must be noted that this specifically happens if the keras.models.load_model function is used.

It appears to be due to a conflict between keras/tensorflow and flask (the underlying web framework used in cortex). More on that here:
keras-team/keras#13353

Alternatives

Alt. 1

One very good way of circumventing this is to replace the

from keras.models import load_model # the one getting the error with

with

from tensorflow.keras.models import load_model # the one with which no error is thrown

Alt. 2

Another possible way to circumvent this is to switch to theano backend by adding KERAS_BACKEND=theano environment variable inside the cortex.yaml configuration file. But this only works for the CPU inferencing. With this backend, to make it work on the GPU too, pygpu has to be installed and this dependecy can only be installed with conda or by building it from source, which is anyway riddled with bugs. Don't forget the theano project is no longer maintained, which means it's only a temporary fix.

Steps to reproduce

1. Deploy an API that loads a keras model (built on 2.3.x) with tensorflow (2.x) using the keras.models.load_model function.
2. Make a prediction.
3. See the error using cortex logs <api name>.

Stack traces

(error output from cortex logs <api name>)

Traceback (most recent call last):
  File "/src/cortex/serve/serve.py", line 86, in predict
    output = predictor_impl.predict(payload)
  File "/mnt/project/predictor.py", line 61, in predict
    pred = self.make_prediction(image)
  File "/mnt/project/predictor.py", line 79, in make_prediction
    pred = self.model.predict(processed)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1452, in predict
    if self._uses_dynamic_learning_phase():
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 382, in _uses_dynamic_learning_phase
    not isinstance(K.learning_phase(), int))
  File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 73, in symbolic_fn_wrapper
    if _SYMBOLIC_SCOPE.value:
AttributeError: '_thread._local' object has no attribute 'value'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/src/cortex/serve/serve.py", line 88, in predict
    raise UserRuntimeException(api.predictor.path, "predict", str(e)) from e

[deliahu]

Thanks for bringing this to our attention! We will be switching from Flask to FastAPI soon (hopefully merging into master in the next few days, and releasing shortly thereafter). Let’s see what happens when that gets in, hopefully the issue will just go away

[deliahu]

@RobertLucian I wonder if this has been resolved now that 0.14 is released, is it easy for you to check this?

[RobertLucian]

@deliahu yeah, I'm adding this to my list of things to do. I will be back with an answer.

[RobertLucian]

@deliahu just tried this on 0.14 and it still doesn't work. Same error. Using from tensorflow.keras.models import load_model still seems to be the best alternative.

[deliahu]

@RobertLucian thank you for checking. Should we close this issue since from tensorflow.keras.models import load_model is working, or is the a change that you would suggest to be implemented in Cortex?

[RobertLucian]

@deliahu I'm thinking the users will most likely look at the examples directory in cortex and see how #834 loads the model with Keras or they'll see this ticket. I think it's okay to close this issue.

[deliahu]

@RobertLucian sounds good, thank you!

[tomercahal]

Hi @RobertLucian Thank you so much!!